Fix launch template reconciliation if bootstrap data secret cannot be read #4589
Conversation
k8s-ci-robot
added
release-note
|
/test ? |
|
@richardcase: The following commands are available to trigger required jobs:
The following commands are available to trigger optional jobs:
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/test pull-cluster-api-provider-aws-e2e |
|
/test pull-cluster-api-provider-aws-e2e-eks |
k8s-ci-robot
added
the
lgtm
|
@richardcase The test |
|
/lgtm |
k8s-ci-robot
added
do-not-merge/contains-merge-commits
and removed
lgtm
|
Trying the test again after having merged /test pull-cluster-api-provider-aws-e2e-eks |
AndiDog
force-pushed
the
machine-pool-bootstrap-config-ref-rotation
branch
from
cbeaab4
to
2b13c74
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/test pull-cluster-api-provider-aws-e2e-eks |
|
/approve |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ankitasw, faiq The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
What type of PR is this?
/kind bug
What this PR does / why we need it:
When changing the machine pool's bootstrap config, e.g.
KubeadmConfig/{a => b}, which is supported after CAPI's fix kubernetes-sigs/cluster-api#8667, CAPA had a bug where it would continue even after the new bootstrap secret cannot be read, for instance because it doesn't exist yet as the bootstrap provider still needs a few seconds to make it ready. That could lead to a catastrophic misconfiguration where the launch template's user data is empty, typically leading to new instances not getting bootstrapped at all. Detailed problem description in this comment.This provides the obvious fix. I couldn't find a great place to test this, since
ReconcileLaunchTemplateis mocked away in tests.I tested this successfully by changing the bootstrap config reference and observing how CAPI sets the new data secret name, CAPA intermittently failing to read the new secret until it's ready, and only then CAPA writing a new launch template version with the new bootstrap secret's data.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Part of kubernetes-sigs/cluster-api#8858, but not a fix. Even with this patch, changing the bootstrap config object doesn't lead to an instance refresh (rolling out new nodes) because CAPA doesn't consider it a change. We need to discuss further how to solve the main issue.
Checklist:
Release note: