Enhancement: Check cloud-init status

[lentzi90]

We recently had a problem in the CI where the OS we use for the Nodes had an update that made one of our preKubeadmCommands fail. This is in turn caused some network issues, which was what we detected first. It took quite some time before we managed to figure out the root cause because no one suspected that there was an error in the cloud-init commands. The Machines were provisioned, the cluster worked as expected in many ways, all Nodes healthy.

My suggestion is that we add a step to the integration tests (or even to the controller if possible) to detect errors in cloud-init and report them in a more obvious way. In the CI we should simply be able to check cloud-init status and error out if it is set to status: error.
To be clear, this check should be done on the workload cluster Nodes.

[Rozzii]

/triage accepted
/kind feature
/help

[Arvinderpal]

CAPI uses a sentinel file to check if bootstrapping succeeded.
kubernetes-sigs/cluster-api#3716

Some providers, like capz, do check this file as an indication of successful bootstrapping.
For baremetal, the tricky part I think is accessing this file from the management cluster.

[Rozzii]

NOTE to everybody: Fell free to work on this and contact/ping @lentzi90 or ask it here if you have any question!

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[metal3-io-bot]

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

[metal3-io-bot]

@metal3-io-bot: Closing this issue.

In response to this:

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[furkatgofurov7]

/reopen
/remove-lifecycle stale

[metal3-io-bot]

@furkatgofurov7: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle stale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[furkatgofurov7]

/remove-help

[furkatgofurov7]

/help

[metal3-io-bot]

@furkatgofurov7:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[Rozzii]

/lifecycle frozen
Would be a good improvement in the future.

[Rozzii]

This would be a nice feature in the kubadm-bootstrap-operator