calling webhook - connection refused

[raminderis]

so i am following the famous kubebuilder book. all went well until i started playing with webhooks to introduce mutliple versions.

when i try to create the cronjob it calls either mutating webhook (if v1) or conversion webhook(if v2).

but both cases i get (this one is when i try to create v1 job):
Error from server (InternalError): error when creating "config/samples/batch_v1_cronjob.yaml": Internal error occurred: failed calling webhook "mcronjob.kb.io": failed to call webhook: Post "https://kubebuilder-webhook-service.kubebuilder-system.svc:443/mutate-batch-tutorial-kubebuilder-io-v1-cronjob?timeout=10s": dial tcp 10.96.169.192:443: connect: connection refused

I see the service, the end points and the kubebuilder-controller-manager pod running normally.

Few Q:

1. am i expected to see a separate webhook pod? on this page https://book.kubebuilder.io/cronjob-tutorial/running-webhook, it says webhook pod will come up, but i only see controller-manager pod with manager and rbac containers. is this correct?
2. any help what i need to do to move past this connection refused error. on the same page at the end I see it talks about bootstrapping problem, i have a feeling somehow i need to use objectSelectors, but when i try to use them in config/webhook/manifests.yaml where WebhookConfiguration is defined, it gets removed when i run "make manifests".

i am stuck here for more then 3 days now. Any help is greatly appreciated.

[camilamacedo86]

Hi @raminderis,

Webhook Server

The webhook server typically runs as a part of the controller-manager pod. The manager container inside this pod is responsible for running both the controllers and the webhook server. See:

kubebuilder/testdata/project-v4/cmd/main.go

Lines 159 to 162 in 2b25ade

	if err = (&crewv1.Admiral{}).SetupWebhookWithManager(mgr); err != nil {
	setupLog.Error(err, "unable to create webhook", "webhook", "Admiral")
	os.Exit(1)
	}

• See in the main.go that the server is created within the Manager:

  kubebuilder/testdata/project-v4/cmd/main.go

  Line 104 in 2b25ade

  WebhookServer: webhookServer,

• See the controller-runtime implementations:

https://github.com/kubernetes-sigs/controller-runtime/blob/479b723944e34ae42c9911fe01228ff34eb5ca81/pkg/manager/manager.go#L244-L247

https://github.com/kubernetes-sigs/controller-runtime/blob/479b723944e34ae42c9911fe01228ff34eb5ca81/pkg/manager/internal.go#L270-L279

Connection Refused Error

The "connection refused" error usually indicates that something is preventing the Kubernetes API server from communicating with your webhook service. Here are a few things you can check and try:

• Did you install the cert-manager?
• Note that typically you will need to have the cert-manager installed and the certs to use within

You can check the patch: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/default/manager_webhook_patch.yaml

You need uncomment:

kubebuilder/testdata/project-v4/config/default/kustomization.yaml

Lines 39 to 142 in 2b25ade

	# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
	# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
	# 'CERTMANAGER' needs to be enabled to use ca injection
	#- path: webhookcainjection_patch.yaml
	# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
	# Uncomment the following replacements to add the cert-manager CA injection annotations
	#replacements:
	# - source: # Add cert-manager annotation to ValidatingWebhookConfiguration, MutatingWebhookConfiguration and CRDs
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# name: serving-cert # this name should match the one in certificate.yaml
	# fieldPath: .metadata.namespace # namespace of the certificate CR
	# targets:
	# - select:
	# kind: ValidatingWebhookConfiguration
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 0
	# create: true
	# - select:
	# kind: MutatingWebhookConfiguration
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 0
	# create: true
	# - select:
	# kind: CustomResourceDefinition
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 0
	# create: true
	# - source:
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# name: serving-cert # this name should match the one in certificate.yaml
	# fieldPath: .metadata.name
	# targets:
	# - select:
	# kind: ValidatingWebhookConfiguration
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 1
	# create: true
	# - select:
	# kind: MutatingWebhookConfiguration
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 1
	# create: true
	# - select:
	# kind: CustomResourceDefinition
	# fieldPaths:
	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
	# options:
	# delimiter: '/'
	# index: 1
	# create: true
	# - source: # Add cert-manager annotation to the webhook Service
	# kind: Service
	# version: v1
	# name: webhook-service
	# fieldPath: .metadata.name # namespace of the service
	# targets:
	# - select:
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# fieldPaths:
	# - .spec.dnsNames.0
	# - .spec.dnsNames.1
	# options:
	# delimiter: '.'
	# index: 0
	# create: true
	# - source:
	# kind: Service
	# version: v1
	# name: webhook-service
	# fieldPath: .metadata.namespace # namespace of the service
	# targets:
	# - select:
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# fieldPaths:
	# - .spec.dnsNames.0
	# - .spec.dnsNames.1
	# options:
	# delimiter: '.'
	# index: 1
	# create: true

PS.: The e2e tests are done with webhooks, you might want to give a look at: https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/v4/plugin_cluster_test.go

Other points to check

• Check if your cluster has not NetworkPolicies or firewalls that are blocking it
• Check the manager logs
• Check the service and endpoints

I hope that answered your question, so I am closing this one.
However, if you need please feel free to re-open it.

Some refs:

• https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation.html