OTEL Collector - error reading server preface: http2: frame too large"

[nnaik25]

Hello,

Need help with the issue i am facing with below usecase

Usecase: Test if we are able to export traces using a local otel collector over ingress to another otel collector service setup on an kubernetes cluster

I want to test this usecase without using self signed certificates.

Also i cannot use CA signed certificates as a part of TLS configuration for the otel collector as this would require CA key and private key of the server ( NGINX in this case) to generate CA signed client certificates which i think is not possible for production environments

Question :

1. Is it a must to use CA signed certificates for TLS configuration of otel collector? If yes, then as i said above, this would require CA key and private key of the server ( NGINX in this case) to generate CA signed client certificates which i think is not possible for productions environments

2. Is there is a way where we can use just the public key of the server ( NGINX in this case) for tls configuration for trusting the server certificate? Similar to how it works in case of browsers

Details

1. Setup a otel-demo environment : https://opentelemetry.io/docs/demo/docker-deployment/

As part of this deployment, otel collector would have been setup as below

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/publicEndpoints: "null"
  creationTimestamp: "2023-04-20T08:32:53Z"
  labels:
    app: my-otel-collector-svc
  name: my-otel-collector-svc
  namespace: iitest
  resourceVersion: "120233132"
  uid: bd7f4b29-eeeb-4966-983e-c72ea2bf318e
spec:
  clusterIP: 10.xx.xx.xx
  clusterIPs:
  - 10.xx.xxx.xxx
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: 4307-4307
    port: 4307
    protocol: TCP
    targetPort: 4307
  selector:
    app: my-otel-collector
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

2. Setup ingress for the otel collector service

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/publicEndpoints: '[{"addresses":["172.xx.xxx.x"],"port":80,"protocol":"HTTP","serviceName":"iitest:my-otel-collector-svc","ingressName":"iitest:otel-grpc-ingress","hostname":"grpcserver","path":"/","allNodes":true}]'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC","nginx.ingress.kubernetes.io/ssl-redirect":"true"},"name":"otel-grpc-ingress","namespace":"iitest"},"spec":{"ingressClassName":"nginx","rules":[{"host":"grpcserver","http":{"paths":[{"backend":{"service":{"name":"my-otel-collector-svc","port":{"number":4307}}},"path":"/","pathType":"Prefix"}]}}]}}
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2023-04-28T14:06:09Z"
  generation: 3
  name: otel-grpc-ingress
  namespace: iitest
  resourceVersion: "124152864"
  uid: 3ee1f008-9352-4e92-bed0-f7867d59bd3c
spec:
  ingressClassName: nginx
  rules:
  - host: grpcserver
    http:
      paths:
      - backend:
          service:
            name: my-otel-collector-svc
            port:
              number: 4307
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 172.xx.xxx.x
    - ip: 172.xx.xxx.x

3. Setup an local otel collector as below

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
  otlp/traces:
     endpoint: grpcserver:443
processors:
  batch:
    send_batch_size: 1
    send_batch_max_size: 1

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/traces]

4. Start the local otel collector

Issue :

Otel collector on a local host is throwing below error when trying to export traces to another otel collector deployed on K8S over ingress

  2023-05-08T12:08:28.784-0500    info    zapgrpc/zapgrpc.go:178  [core] [Server #3 ListenSocket #4] ListenSocket created {"grpc_log": true}
2023-05-08T12:08:29.781-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: closing: connection error: desc = "error reading server preface: http2: frame too large" {"grpc_log": true}
2023-05-08T12:08:29.784-0500    info    zapgrpc/zapgrpc.go:178  [transport] transport: loopyWriter exited. Closing connection. Err: transport closed by client  {"grpc_log": true}
2023-05-08T12:08:29.784-0500    info    zapgrpc/zapgrpc.go:178  [core] Creating new client transport to "{\n  \"Addr\": \"grpcserver:443\",\n  \"ServerName\": \"grpcserver:443\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "error reading server preface: http2: frame too large"
{"grpc_log": true}
2023-05-08T12:08:29.792-0500    warn    zapgrpc/zapgrpc.go:195  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "grpcserver:443",
  "ServerName": "grpcserver:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "error reading server preface: http2: frame too large" {"grpc_log": true}

Debugging Steps Tried:

1. Tried setting below parameter in nginx configuration config map. But error still exists
   proxy_buffers: 4 16k

[addname]

I have the same problem and don't know how to solve it

[shubhwip]

Problems on similar line, don't know how to solve it.
https://stackoverflow.com/questions/76386139/opentelemetry-collector-not-exporting-data-to-otel-http-exporter

[atoulme]

This error:

}. Err: connection error: desc = "error reading server preface: http2: frame too large" {"grpc_log": true}

says that you are sending a HTTP request to a gRPC endpoint. Make sure to enable the http protocol.

[diptripa]

+1

[mk-raven]

+1

[digdiego13]

+1

[AnomalRoil]

This error is usually a symptom of trying to connect without TLS to a server that's expecting TLS. Are you dialing these connections using TLS or not?

[keoren3]

I had a similar issue and just solved it.
In the local OTEL collector you're passing the GRPC through port 443.
Port 443 won't work with GRPC (I'm not sure why, I think it's related to the fact that it's a well known port) and it didn't work for me, and I got the http2 error that you got.

HOW DID I SOLVE THIS:
So first of all don't send the GRPC data through port 443, instead choose a different one - I use Uptrace, so I used port 14317 (I can see you chose 4307)

On the Ingress side, don't receive the data through 443, but instead design a designate port (4307).
I did it by adding the next annotation to my Ingress:
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 14318}, {"HTTPS": 14317}]'
(14318 is just another UI port I needed)

Which does use an SSL certificate, and forwards to port 14317 (You should use 4307).

You can see that I use EKS, and my load-balancer is in AWS too, so you'll need to make some adjustments.
Once you send the data through 4307 from the OTEL collector, to port 4307 in the Ingress, it should work as expected.

[Gihchathur]

This is happening because you're using the OTLP exporter, which is a gRPC exporter. However, you need an HTTP endpoint, so you'll likely need to update the OTLP exporter to otlphttp.