Unable to find pod with katalog-sync annotation (katalog-sync.wish.com/service-names)

[ylohnitram]

time="2020-03-11T10:45:36Z" level=error msg="error registering with katalog-sync-daemon: Unknown rpc error: code = Unknown desc = Unable to find pod with katalog-sync annotation (katalog-sync.wish.com/service-names): /api/v1/namespaces/default/pods/hw-7c88d77f6-99c2m"

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
name: hw
labels:
app: hw
annotations:
katalog-sync.wish.com/sidecar: "katalog-sync-sidecar"
katalog-sync.wish.com/service-names: "hw"
katalog-sync.wish.com/service-port: '8080'
spec:
replicas: 1
selector:
matchLabels:
app: hw
template:
metadata:
labels:
app: hw
annotations:
"katalog-sync.wish.com/sidecar": "katalog-sync-sidecar"
"katalog-sync.wish.com/service-names": "hw"
"katalog-sync.wish.com/service-port": '8080'
#katalog-sync.wish.com/service-port-servicename: '12345'
#katalog-sync.wish.com/service-meta: 'a:1,b:2'
#katalog-sync.wish.com/service-meta-servicename: 'b:1,c:2'
#katalog-sync.wish.com/service-tags: a,b
"katalog-sync.wish.com/sync-interval": 2s
spec:
terminationGracePeriodSeconds: 1
containers:
- name: hw
image: "... IMAGE ..."
imagePullPolicy: Always
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: "/live"
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: "/ready"
port: 8080
periodSeconds: 5
- command:
- "/bin/katalog-sync-sidecar"
args:
- "--katalog-sync-daemon=$(HOST_IP):8501"
- "--namespace=$(MY_POD_NAMESPACE)"
- "--pod-name=$(MY_POD_NAME)"
- "--container-name=katalog-sync-sidecar"
- "--bind-address=:8888"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: "... IMAGE ..."
imagePullPolicy: Always
name: katalog-sync-sidecar
readinessProbe:
httpGet:
path: "/ready"
port: 8080
initialDelaySeconds: 1
periodSeconds: 5

[jacksontj]

Curious as this looks like the example configuration (although the indentation got eaten by github). Could you re-paste in a code block (using ` 3 times)-- as it looks like it should work.

[jacksontj]

I'm going to close this out for now as we haven't heard back in a while. If you are still seeing the same issue please feel free to re-open :)

[gempesaw]

i got this error as well, but it was a slightly different culprit. as part of our hardened AWS EKS image, we disable the kubelet readonly port that is typically open on 10255, so we have to use the read/write https port with authentication on https://localhost:10250.

awslabs/amazon-eks-ami#128

        env:
        - name: KUBELET_API
          value: 'https://$(HOST_IP):10250/pods'

(also i didn't use hostNetwork: true so that's why i'm using $(HOST_IP) injected via downwardAPI. for hostNetwork: true, it should be fine for https://localhost:10250/pods)

---

i did have some trouble debugging at first, cuz i read the docs wrong and didn't pass kubelet address?

instead i passed the normal like .. https://kubernetes.default:443/pods kubernetes API default namespace Service address. but of course that is not as helpful, because it's not the node-local pods that you get from the kubelet. and also /pods is not a valid URL in the kubernetes API lol

but the trick was, neither the sidecar nor the daemonset indicated that i did anything wrong, other than the error message unable to find pod...