it seems TLS not working for OTLP receiver with gRPC

[chriske]

Describe the bug
I'm using OpenTelemetry Operator to provide Collectors and Java Instrumentation injection for applications. In this scenario the Java otel agent will be inject into a pod, and I try to connect it to the Collector with TLS or mTLS through gRPC. I tried several configuration variations, but I'm getting this error in the Collector whatever I try (in debug logging):

info zapgrpc/zapgrpc.go:176	[transport] [server-transport 0xc000007a00] Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\""	{"grpc_log": true}

info	zapgrpc/zapgrpc.go:176	[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\""	{"grpc_log": true}

When I switch the comunication with the same certificates to HTTP/protobuf, everything works fine.

Steps to reproduce
I'm using cert-manager to create my CA and all of the certificates with these settings:

volumeAttributes:<br>
              csi.cert-manager.io/issuer-name: otel-ca-issuer
              csi.cert-manager.io/issuer-kind: ClusterIssuer
              csi.cert-manager.io/dns-names: ${POD_NAME}.default.svc.cluster.local
              csi.cert-manager.io/key-usages: client auth,server auth,key encipherment,digital signature
              csi.cert-manager.io/key-encoding: PKCS8
              csi.cert-manager.io/common-name: java-app

This is how my Instrumentation config looks like:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation-sample
spec:
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"  
  java:    
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: https://central-collector-collector.monitoring.svc.cluster.local:4317   
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: grpc
      - name: OTEL_EXPORTER_OTLP_CERTIFICATE
        value: /tls/ca.crt
      - name: OTEL_EXPORTER_OTLP_CLIENT_KEY
        value: /tls/tls.key
      - name: OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE
        value: /tls/tls.crt

On the other side, here is how the Collector configuration looks like:

receivers:      
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317            
            tls:                        
              ca_file: /tls/ca.crt    
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key
              client_ca_file: /tls/ca.crt       
          http:
            tls:
              ca_file: /tls/ca.crt    
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key

As I told before, if I switch my configuration in the Instrumentation object to use HTTP instead of gRPC, everything works fine.

Now, I tried another scenario: I chained two Collectors with the same settings and certificates on gRPC. That connection works fine too. Here is how I did it:

Collector config on the "edge" collector:

exporters:      
      otlp/http:
          endpoint: https://central-collector-remote-collector.monitoring.svc.cluster.local:4317
          tls:
            insecure: false
            ca_file: /tls/ca.crt
            cert_file: /tls/tls.crt
            key_file: /tls/tls.key
            min_version: "1.1"
            max_version: "1.2" 

And here is it's pair on the "remote" collector:

receivers:      
      otlp:
        protocols:
          grpc:
            tls:
              ca_file: /tls/ca.crt
              cert_file: /tls/tls.crt
              key_file: /tls/tls.key

This configuration also works between the two collectors, they can communicate with TLS on gRPC.

What did you expect to see?
Working TLS with gRPC.

What did you see instead?

info zapgrpc/zapgrpc.go:176	[transport] [server-transport 0xc000007a00] Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\""	{"grpc_log": true}

info	zapgrpc/zapgrpc.go:176	[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\""	{"grpc_log": true}

What version did you use?
Collector: v.0.96.0 contrib
Java autoinstrumentation image: 1.32.1

Environment
Kubernetes v1.27

[TylerHelmuth]

Now, I tried another scenario: I chained two Collectors with the same settings and certificates on gRPC. That connection works fine too.

Based on this, I think the problem is with Java and not the collector. My guess is that the Java agent is correctly using the env vars to setup TLS connection over grpc. I am very suspicious of this part of the error message \"POST /v1/metrics HTTP/1.\"".