Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 cleanup: eliminate log spam when using S3 secrets #4667

Merged
merged 1 commit into from
Jan 11, 2024
Merged

🐛 cleanup: eliminate log spam when using S3 secrets #4667

merged 1 commit into from
Jan 11, 2024

Conversation

thefirstofthe300
Copy link
Contributor

@thefirstofthe300 thefirstofthe300 commented Nov 29, 2023 •
edited
Loading

/kind cleanup

Currently, the S3 object store generates a large amount of log spam during reconciliation. Objects are stated to be deleting regardless of objects already being deleted.

Example:

I1129 02:23:28.008345       1 logger.go:67] "Deleting unneeded entry from AWS S3" secretPrefix=""
I1129 02:23:28.008394       1 logger.go:67] "Deleting object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="clusters-system/us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk" namespace="clusters-system" name="us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk" reconcileID="c10cc649-57ce-40f4-99bb-17d06590c2cf" machine="clusters-system/us-west-2-production-worker-us-west-2d-5lgbh-6cbb9999d5xlnxtcvd" cluster="clusters-system/us-west-2-production" bucket_name="cluster-api-provider-aws-gremlin-us-west-2-production" key="node/us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk"
I1129 02:24:05.691604       1 awsmanagedcluster_controller.go:114] "Successfully reconciled AWSManagedCluster" controller="awsmanagedcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSManagedCluster" AWSManagedCluster="clusters-system/us-east-1-production-testing" namespace="clusters-system" name="us-east-1-production-testing" reconcileID="8cbaa551-85ef-4feb-9fac-533577193662" cluster="us-east-1-production-testing" controlPlane="us-east-1-production-testing-control-plane"
  • squashed commits
  • includes documentation (N/A)
  • includes emojis
  • adds unit tests (N/A
  • adds or updates e2e tests

Release note:

If you're using S3 buckets for bootstrap data, the following permissions are required on the controller to automatically detect whether an object is still needing to be deleted: `s3:GetObject`. The following permissions are optional: `s3:ListBucket`.

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority labels Nov 29, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @thefirstofthe300. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 29, 2023
AndiDog
AndiDog reviewed Nov 29, 2023
View reviewed changes
pkg/cloud/services/s3/s3.go Show resolved Hide resolved
pkg/cloud/services/s3/s3_test.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 29, 2023
@vincepri
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 29, 2023
@vincepri
Copy link
Member

/test ?

@k8s-ci-robot
Copy link
Contributor

@vincepri: The following commands are available to trigger required jobs:

  • /test pull-cluster-api-provider-aws-build
  • /test pull-cluster-api-provider-aws-test
  • /test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

  • /test pull-cluster-api-provider-aws-apidiff-main
  • /test pull-cluster-api-provider-aws-e2e
  • /test pull-cluster-api-provider-aws-e2e-blocking
  • /test pull-cluster-api-provider-aws-e2e-clusterclass
  • /test pull-cluster-api-provider-aws-e2e-conformance
  • /test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
  • /test pull-cluster-api-provider-aws-e2e-eks
  • /test pull-cluster-api-provider-aws-e2e-eks-gc
  • /test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

  • pull-cluster-api-provider-aws-apidiff-main
  • pull-cluster-api-provider-aws-build
  • pull-cluster-api-provider-aws-test
  • pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@vincepri
Copy link
Member

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e-eks-gc
/test pull-cluster-api-provider-aws-e2e-eks-testing

AndiDog
AndiDog reviewed Dec 8, 2023
View reviewed changes
Copy link
Contributor

@AndiDog AndiDog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One compatibility blocker. Rest looks fine.

pkg/cloud/services/s3/s3.go Outdated Show resolved Hide resolved
AndiDog added a commit to giantswarm/giantswarm-aws-account-prerequisites that referenced this pull request Dec 8, 2023
@AndiDog
Add s3:GetObject permission for CAPA
a8d084c 
For upcoming feature kubernetes-sigs/cluster-api-provider-aws#4667
@AndiDog AndiDog mentioned this pull request Dec 8, 2023
Merged
AndiDog added a commit to giantswarm/giantswarm-aws-account-prerequisites that referenced this pull request Dec 8, 2023
@AndiDog
Add s3:GetObject permission for CAPA
222b2f6 
For upcoming feature kubernetes-sigs/cluster-api-provider-aws#4667
AndiDog added a commit to giantswarm/giantswarm-aws-account-prerequisites that referenced this pull request Dec 11, 2023
@AndiDog
Add s3:GetObject permission for CAPA (#85)
ee7bbf7 
For upcoming feature kubernetes-sigs/cluster-api-provider-aws#4667
@thefirstofthe300
Copy link
Contributor Author

/retest

@Ankitasw
Copy link
Member

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@Ankitasw
Copy link
Member

/approve

cc @AndiDog for another review

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ankitasw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 11, 2024
@AndiDog
Copy link
Contributor

AndiDog commented Jan 11, 2024

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2024
@k8s-ci-robot k8s-ci-robot merged commit 2234387 into kubernetes-sigs:main Jan 11, 2024
19 checks passed
@thefirstofthe300 thefirstofthe300 deleted the reduce-log-spam branch January 11, 2024 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@faiq faiq Awaiting requested review from faiq

@vincepri vincepri Awaiting requested review from vincepri

@AndiDog AndiDog Awaiting requested review from AndiDog

Assignees

@AndiDog AndiDog

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@thefirstofthe300 @k8s-ci-robot @vincepri @Ankitasw @AndiDog