Incorrect state in health check event

[phillebaba]

When deploying a Kustomization with a health check to a deployment in an incorrect image tag, the expected event generated is expected to be an error. The image cant be pulled which means that the deployment will fail immediately and never reach a healthy state. The correct error event is received if it is the initial deployment or if the previous commit also caused the health check to fail.

If a deployment is healthy but then a new commit changes the tag to something that cant be fetched, the health check will transition from healthy in one commit to unhealthy in another commit. The first event will have the wrong severity but the correct reason. After the next reconcile loop is run for the kustomization the health check will be evaluated again and the correct severity will be sent. I have only observed this issue in the git commit notifier, and it may only affect those notifiers as they will not ignore update events sent by the kustomize controller.

Below is the status messages sent for a commit to GitHub

[
  {
    "url": "",
    "avatar_url": "",
    "id": 0,
    "node_id": "",
    "state": "failure",
    "description": "health check failed",
    "target_url": null,
    "context": "kustomization/apps",
    "created_at": "2020-12-04T11:18:19Z",
    "updated_at": "2020-12-04T11:18:19Z",
    "creator": {}
  },
  {
    "url": "",
    "avatar_url": "",
    "id": 0,
    "node_id": "",
    "state": "success",
    "description": "health check failed",
    "target_url": null,
    "context": "kustomization/apps",
    "created_at": "2020-12-04T11:17:18Z",
    "updated_at": "2020-12-04T11:17:18Z",
    "creator": {}
  }
]

Thank you @chlunde for initially reporting this bug and helping out with digging into it!