Prometheus receiver metric_label_configs not working from label

[ckt114]

Component(s)

cmd/otelcontribcol

What happened?

Description

When using the metric_relabel_configs to rename metrics using value from annotation it doesn't work. It does work when using string literal.

Steps to Reproduce

1. Add annotation to Kubernetes Service

   apiVersion: v1
   kind: Service
   metadata:
      name: service-01
   annotations:
      [...]
      prometheus.io/metrics-prefix: "testprefix"

2. Configure prometheus receiver

   scrape_configs:
   - job_name: kubernetes-services
   [...]
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metrics_prefix]
     target_label: metrics_prefix
     action: replace
   
   # This does not work
   metric_relabel_configs:
   - source_labels: [metrics_prefix, __name__]
     regex: (.+);(.+)
     target_label: __name__
     replacement: $1_$2
     action: replace
   
   # But this works
   metric_relabel_configs:
   - source_labels: [metrics_prefix, __name__]
     regex: (.+);(.+)
     target_label: __name__
     replacement: foo_$2
     action: replace

NOTES: I've also tried replacement with values ${1}_${2} and $$1_$$2 but it still doesn't work. I wrote the metrics to file using file exporter and I did see that metrics_prefix label does have the value testprefix in the file.

Expected Result

Each metric name, except for up, should be prefixed with testprefix_.

Actual Result

All metrics names stay the same.

Collector version

v0.95.0

Environment information

Environment

OS: ContainerOS
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Provided above.

Log output

No response

Additional context

No response

[ckt114]

Finally tried replacement: $${1}_$${2} and it worked.