why does the target_info still have values after a pod is evicted ？

[wangjinxiang0522]

Component(s)

receiver/prometheus
exporter/prometheusremotewrite

Describe the issue you're reporting

    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
                - targets: ['0.0.0.0:8888']
        target_allocator:
          endpoint: http://mc-collector-ta-targetallocator
          interval: 30s
          collector_id: "${POD_NAME}"
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      logging:
        #loglevel: debug
      prometheusremotewrite:
        endpoint: http://mimir-nginx.monitoring.svc:80/api/v1/push
        tls:
          insecure: true
        resource_to_telemetry_conversion:
          enabled: false

Actual Result

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Does it stick around for about 5 mintues? If so, this sounds like we are missing staleness markers when a pod is evicted.

[wangjinxiang0522]

Does it stick around for about 5 mintues? If so, this sounds like we are missing staleness markers when a pod is evicted.

@dashpole Yes，thanks for your reply, how should I modify the parameters to solve this issue?

[dashpole]

Does this happen only when a pod is evicted? Or also when a pod is deleted?

[wangjinxiang0522]

Does this happen only when a pod is evicted? Or also when a pod is deleted?

Yes, it happens when a pod is evicted or deleted.

[dashpole]

my best guess is that when we apply the new config to the service manager and discovery manager, it removes the targets without generating a staleness marker. But fixing it will probably be a change in the prometheus server (promethues/prometheus). We need to reproduce this with the prometheus server (updating the config file to remove a static target), and see if the series is marked stale (if the line correctly stops) in the graph.