receiver/kubeletstats cannot read /pods with nodes/proxy permissions

[Joibel]

Component(s)

receiver/kubeletstats

What happened?

Description

If I use kubeletstats with a clusterrole with

- apiGroups:
    - ""
  resources:
    - nodes/stats
    - nodes/proxy
  verbs:
    - get

and with a receiver config of

              extra_metadata_labels:                                                                                                                                                                                                           
                - container.id   

I get

2024-01-03T11:09:05.622Z    error    scraperhelper/scrapercontroller.go:200    Error scraping metrics    {"kind": "receiver", "name": "receiver_creator/kubeletstats", "data_type": "metrics", "name": "kubeletstats//receiver_creator/kubeletstats{endpoint=\"172.20.24.255:10250\"}/k8s_observer/ip-10-106-248-44.us-east-2.compute.internal-1c3ebbc5-a1b7-494f-b2a0-70a938e1f782", "error": "kubelet request GET https://172.20.24.255:10250/pods failed - \"403 Forbidden\", response: \"{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"forbidden: User \\\\\\\"system:serviceaccount:sample:sample-collector\\\\\\\" cannot get path \\\\\\\"/pods\\\\\\\"\\\",\\\"reason\\\":\\\"Forbidden\\\",\\\"details\\\":{},\\\"code\\\":403}\\n\"", "scraper": "kubeletstats"}    

If I add

- nonResourceURLs:
    - /pods
  verbs:
    - get

then the logged error goes away (and removing nodes/proxy has no effect)

Expected Result

Successful scraping of /pods according to the readme

Actual Result

Failure to read /pods

Collector version

0.91.0

Environment information

Environment

Kubernetes 1.28.4 on Arm.

OpenTelemetry Collector configuration

receivers:
      kubeletstats:                                                                                                                                                                                                                                                                                                                                                                      
        auth_type: serviceAccount                                                                                                                                                                                                        
        collection_interval: 10s                                                                                                                                                                                                         
        endpoint: "${env:K8S_NODE_NAME}:10250"                                                                                                                                                                                   
          extra_metadata_labels:                                                                                                                                                                                                           
            - container.id                                                                                                                                                                                                                 
          metric_groups:                                                                                                                                                                                                                   
            - container                                                                                                                                                                                                                    
            - pod                                                                                                                                                                                                                          
            - node                                                                                                                                                                                                                         
                                                                                                                                                                                                                                               
    exporters:                                                                                                                                                                                                                                 
      otlp:                                                                                                                                                                                                                                    
        endpoint: collector:4317                                                                                                                                                                                               
                                                                                                                                                                                                                                               
    service:                                                                                                                                                                                                                                   
      pipelines:                                                                                                                                                                                                                               
        metrics:                                                                                                                                                                                                                               
          receivers: [kubeletstats]                                                                                                                                                            
          exporters: [otlp]                                                                                                                                                                                                   
      telemetry:                                                                                                                                                                                                                               
        logs:                                                                                                                                                                                                                                  
          level: debug                                                                                                                                                                                                                         
        metrics:                                                                                                                                                                                                                               
          address: ":8888"

Log output

2024-01-03T11:09:05.622Z    error    scraperhelper/scrapercontroller.go:200    Error scraping metrics    {"kind": "receiver", "name": "receiver_creator/kubeletstats", "data_type": "metrics", "name": "kubeletstats//receiver_creator/kubeletstats{endpoint=\"172.20.24.255:10250\"}/k8s_observer/ip-10-106-248-44.us-east-2.compute.internal-1c3ebbc5-a1b7-494f-b2a0-70a938e1f782", "error": "kubelet request GET https://172.20.24.255:10250/pods failed - \"403 Forbidden\", response: \"{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"forbidden: User \\\\\\\"system:serviceaccount:sample:sample-collector\\\\\\\" cannot get path \\\\\\\"/pods\\\\\\\"\\\",\\\"reason\\\":\\\"Forbidden\\\",\\\"details\\\":{},\\\"code\\\":403}\\n\"", "scraper": "kubeletstats"}

Additional context

Removing nodes/stats from the cluster role gives a similar error from the path /stats/summary so that part of the cluster role is working as expected.

[github-actions]

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@Joibel your error message implies you are using the receiver creator, is that correct? If you do not use the receiver creator do you still experience an issue?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.