From 4d1a07b187424d5a1987411e2d454e55bcf6e25a Mon Sep 17 00:00:00 2001 From: jack-berg Date: Sat, 4 Jun 2022 13:03:53 -0500 Subject: [PATCH 01/17] Add kafka client metrics --- instrumentation/kafka/kafka-clients/README.md | 229 +++++++++++++++++- .../testing/build.gradle.kts | 5 + .../OpenTelemetryKafkaMetricsTest.java | 98 ++++++++ .../kafkaclients/MetricsTest.java | 4 + .../kafkaclients/KafkaMetricId.java | 100 ++++++++ .../kafkaclients/KafkaMetricRegistry.java | 105 ++++++++ .../kafkaclients/MetricDescriptor.java | 23 ++ .../OpenTelemetryKafkaMetrics.java | 201 +++++++++++++++ .../kafkaclients/RegisteredInstrument.java | 16 ++ 9 files changed, 780 insertions(+), 1 deletion(-) create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java diff --git a/instrumentation/kafka/kafka-clients/README.md b/instrumentation/kafka/kafka-clients/README.md index 170efde43cf8..f027e0c895f8 100644 --- a/instrumentation/kafka/kafka-clients/README.md +++ b/instrumentation/kafka/kafka-clients/README.md @@ -3,4 +3,231 @@ | System property | Type | Default | Description | |---|---|---|---| | `otel.instrumentation.kafka.experimental-span-attributes` | Boolean | `false` | Enable the capture of experimental span attributes. | -| `otel.instrumentation.kafka.client-propagation.enabled` | Boolean | `true` | Enables remote context propagation via Kafka message headers. | \ No newline at end of file +| `otel.instrumentation.kafka.client-propagation.enabled` | Boolean | `true` | Enables remote context propagation via Kafka message headers. | + +# Kafka Metrics + +The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface. +OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry. + +To use, configure `OpenTelemetryKafkaMetrics` with an OpenTelemetry instance +via `OpenTelemetryKafkaMetrics#setOpenTelemetry(OpenTelemetry)`, and include a reference to this +class in kafka producer or consumer configuration, i.e.: + +```java +Map config = new HashMap<>(); +config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); +config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); +... +try (KafkaProducer producer = new KafkaProducer<>(config)) { ... } +``` + +The following table shows the full set of metrics exposed by the kafka client, and the corresponding +OpenTelemetry metric each maps to (if available). Empty values in the Instrument Name, Instrument +Description, etc column indicates there is no registered mapping for the metric and data is NOT +collected. + +| Kafka Group | Kafka Name | Kafka Description | Attribute Keys | Instrument Name | Instrument Description | Instrument Unit | Instrument Type | +|-------------|------------|-------------------|----------------|-----------------|------------------------|-----------------|-----------------| +| app-info | commit-id | Metric indicating commit-id | client-id | | | | | +| app-info | start-time-ms | Metric indicating start-time-ms | client-id | | | | | +| app-info | version | Metric indicating version | client-id | | | | | +| consumer-coordinator-metrics | assigned-partitions | The number of partitions currently assigned to this consumer | client-id | | | | | +| consumer-coordinator-metrics | commit-latency-avg | The average time taken for a commit request | client-id | | | | | +| consumer-coordinator-metrics | commit-latency-max | The max time taken for a commit request | client-id | | | | | +| consumer-coordinator-metrics | commit-rate | The number of commit calls per second | client-id | | | | | +| consumer-coordinator-metrics | commit-total | The total number of commit calls | client-id | | | | | +| consumer-coordinator-metrics | failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | client-id | | | | | +| consumer-coordinator-metrics | failed-rebalance-total | The total number of failed rebalance events | client-id | | | | | +| consumer-coordinator-metrics | heartbeat-rate | The number of heartbeats per second | client-id | | | | | +| consumer-coordinator-metrics | heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | client-id | | | | | +| consumer-coordinator-metrics | heartbeat-total | The total number of heartbeats | client-id | | | | | +| consumer-coordinator-metrics | join-rate | The number of group joins per second | client-id | | | | | +| consumer-coordinator-metrics | join-time-avg | The average time taken for a group rejoin | client-id | | | | | +| consumer-coordinator-metrics | join-time-max | The max time taken for a group rejoin | client-id | | | | | +| consumer-coordinator-metrics | join-total | The total number of group joins | client-id | | | | | +| consumer-coordinator-metrics | last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | client-id | | | | | +| consumer-coordinator-metrics | last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | client-id | | | | | +| consumer-coordinator-metrics | partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | client-id | | | | | +| consumer-coordinator-metrics | rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | client-id | | | | | +| consumer-coordinator-metrics | rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | client-id | | | | | +| consumer-coordinator-metrics | rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | client-id | | | | | +| consumer-coordinator-metrics | rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | client-id | | | | | +| consumer-coordinator-metrics | rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | client-id | | | | | +| consumer-coordinator-metrics | sync-rate | The number of group syncs per second | client-id | | | | | +| consumer-coordinator-metrics | sync-time-avg | The average time taken for a group sync | client-id | | | | | +| consumer-coordinator-metrics | sync-time-max | The max time taken for a group sync | client-id | | | | | +| consumer-coordinator-metrics | sync-total | The total number of group syncs | client-id | | | | | +| consumer-fetch-manager-metrics | bytes-consumed-rate | The average number of bytes consumed per second | client-id | | | | | +| consumer-fetch-manager-metrics | bytes-consumed-rate | The average number of bytes consumed per second for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | bytes-consumed-total | The total number of bytes consumed for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | bytes-consumed-total | The total number of bytes consumed | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-latency-avg | The average time taken for a fetch request. | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-latency-max | The max time taken for any fetch request. | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-rate | The number of fetch requests per second. | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-size-avg | The average number of bytes fetched per request | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-size-avg | The average number of bytes fetched per request for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | fetch-size-max | The maximum number of bytes fetched per request for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | fetch-size-max | The maximum number of bytes fetched per request | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-throttle-time-avg | The average throttle time in ms | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-throttle-time-max | The maximum throttle time in ms | client-id | | | | | +| consumer-fetch-manager-metrics | fetch-total | The total number of fetch requests. | client-id | | | | | +| consumer-fetch-manager-metrics | preferred-read-replica | The current read replica for the partition, or -1 if reading from leader | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-consumed-rate | The average number of records consumed per second | client-id | | | | | +| consumer-fetch-manager-metrics | records-consumed-rate | The average number of records consumed per second for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | records-consumed-total | The total number of records consumed for a topic | client-id,topic | | | | | +| consumer-fetch-manager-metrics | records-consumed-total | The total number of records consumed | client-id | | | | | +| consumer-fetch-manager-metrics | records-lag | The latest lag of the partition | client-id,topic,partition | messaging.kafka.consumer.lag | Current approximate lag of consumer group at partition of topic. | {lag} | DOUBLE_OBSERVABLE_GAUGE | +| consumer-fetch-manager-metrics | records-lag-avg | The average lag of the partition | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-lag-max | The maximum lag in terms of number of records for any partition in this window | client-id | | | | | +| consumer-fetch-manager-metrics | records-lag-max | The max lag of the partition | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-lead | The latest lead of the partition | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-lead-avg | The average lead of the partition | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-lead-min | The minimum lead in terms of number of records for any partition in this window | client-id | | | | | +| consumer-fetch-manager-metrics | records-lead-min | The min lead of the partition | client-id,topic,partition | | | | | +| consumer-fetch-manager-metrics | records-per-request-avg | The average number of records in each request | client-id | | | | | +| consumer-fetch-manager-metrics | records-per-request-avg | The average number of records in each request for a topic | client-id,topic | | | | | +| consumer-metrics | connection-close-rate | The number of connections closed per second | client-id | | | | | +| consumer-metrics | connection-close-total | The total number of connections closed | client-id | | | | | +| consumer-metrics | connection-count | The current number of active connections. | client-id | | | | | +| consumer-metrics | connection-creation-rate | The number of new connections established per second | client-id | | | | | +| consumer-metrics | connection-creation-total | The total number of new connections established | client-id | | | | | +| consumer-metrics | failed-authentication-rate | The number of connections with failed authentication per second | client-id | | | | | +| consumer-metrics | failed-authentication-total | The total number of connections with failed authentication | client-id | | | | | +| consumer-metrics | failed-reauthentication-rate | The number of failed re-authentication of connections per second | client-id | | | | | +| consumer-metrics | failed-reauthentication-total | The total number of failed re-authentication of connections | client-id | | | | | +| consumer-metrics | incoming-byte-rate | The number of bytes read off all sockets per second | client-id | | | | | +| consumer-metrics | incoming-byte-total | The total number of bytes read off all sockets | client-id | | | | | +| consumer-metrics | io-ratio | The fraction of time the I/O thread spent doing I/O | client-id | | | | | +| consumer-metrics | io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | client-id | | | | | +| consumer-metrics | io-wait-ratio | The fraction of time the I/O thread spent waiting | client-id | | | | | +| consumer-metrics | io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | client-id | | | | | +| consumer-metrics | io-waittime-total | The total time the I/O thread spent waiting | client-id | | | | | +| consumer-metrics | iotime-total | The total time the I/O thread spent doing I/O | client-id | | | | | +| consumer-metrics | last-poll-seconds-ago | The number of seconds since the last poll() invocation. | client-id | | | | | +| consumer-metrics | network-io-rate | The number of network operations (reads or writes) on all connections per second | client-id | | | | | +| consumer-metrics | network-io-total | The total number of network operations (reads or writes) on all connections | client-id | | | | | +| consumer-metrics | outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | client-id | | | | | +| consumer-metrics | outgoing-byte-total | The total number of outgoing bytes sent to all servers | client-id | | | | | +| consumer-metrics | poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | client-id | | | | | +| consumer-metrics | reauthentication-latency-avg | The average latency observed due to re-authentication | client-id | | | | | +| consumer-metrics | reauthentication-latency-max | The max latency observed due to re-authentication | client-id | | | | | +| consumer-metrics | request-rate | The number of requests sent per second | client-id | | | | | +| consumer-metrics | request-size-avg | The average size of requests sent. | client-id | | | | | +| consumer-metrics | request-size-max | The maximum size of any request sent. | client-id | | | | | +| consumer-metrics | request-total | The total number of requests sent | client-id | | | | | +| consumer-metrics | response-rate | The number of responses received per second | client-id | | | | | +| consumer-metrics | response-total | The total number of responses received | client-id | | | | | +| consumer-metrics | select-rate | The number of times the I/O layer checked for new I/O to perform per second | client-id | | | | | +| consumer-metrics | select-total | The total number of times the I/O layer checked for new I/O to perform | client-id | | | | | +| consumer-metrics | successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | client-id | | | | | +| consumer-metrics | successful-authentication-rate | The number of connections with successful authentication per second | client-id | | | | | +| consumer-metrics | successful-authentication-total | The total number of connections with successful authentication | client-id | | | | | +| consumer-metrics | successful-reauthentication-rate | The number of successful re-authentication of connections per second | client-id | | | | | +| consumer-metrics | successful-reauthentication-total | The total number of successful re-authentication of connections | client-id | | | | | +| consumer-metrics | time-between-poll-avg | The average delay between invocations of poll(). | client-id | | | | | +| consumer-metrics | time-between-poll-max | The max delay between invocations of poll(). | client-id | | | | | +| consumer-node-metrics | incoming-byte-rate | The number of incoming bytes per second | client-id,node-id | | | | | +| consumer-node-metrics | incoming-byte-total | The total number of incoming bytes | client-id,node-id | | | | | +| consumer-node-metrics | outgoing-byte-rate | The number of outgoing bytes per second | client-id,node-id | | | | | +| consumer-node-metrics | outgoing-byte-total | The total number of outgoing bytes | client-id,node-id | | | | | +| consumer-node-metrics | request-latency-avg | | client-id,node-id | | | | | +| consumer-node-metrics | request-latency-max | | client-id,node-id | | | | | +| consumer-node-metrics | request-rate | The number of requests sent per second | client-id,node-id | | | | | +| consumer-node-metrics | request-size-avg | The average size of requests sent. | client-id,node-id | | | | | +| consumer-node-metrics | request-size-max | The maximum size of any request sent. | client-id,node-id | | | | | +| consumer-node-metrics | request-total | The total number of requests sent | client-id,node-id | | | | | +| consumer-node-metrics | response-rate | The number of responses received per second | client-id,node-id | | | | | +| consumer-node-metrics | response-total | The total number of responses received | client-id,node-id | | | | | +| kafka-metrics-count | count | total number of registered metrics | client-id | | | | | +| producer-metrics | batch-size-avg | The average number of bytes sent per partition per-request. | client-id | | | | | +| producer-metrics | batch-size-max | The max number of bytes sent per partition per-request. | client-id | | | | | +| producer-metrics | batch-split-rate | The average number of batch splits per second | client-id | | | | | +| producer-metrics | batch-split-total | The total number of batch splits | client-id | | | | | +| producer-metrics | buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | client-id | | | | | +| producer-metrics | buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | client-id | | | | | +| producer-metrics | buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | client-id | | | | | +| producer-metrics | buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | client-id | | | | | +| producer-metrics | bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | client-id | | | | | +| producer-metrics | bufferpool-wait-time-total | The total time an appender waits for space allocation. | client-id | | | | | +| producer-metrics | compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | client-id | | | | | +| producer-metrics | connection-close-rate | The number of connections closed per second | client-id | | | | | +| producer-metrics | connection-close-total | The total number of connections closed | client-id | | | | | +| producer-metrics | connection-count | The current number of active connections. | client-id | | | | | +| producer-metrics | connection-creation-rate | The number of new connections established per second | client-id | | | | | +| producer-metrics | connection-creation-total | The total number of new connections established | client-id | | | | | +| producer-metrics | failed-authentication-rate | The number of connections with failed authentication per second | client-id | | | | | +| producer-metrics | failed-authentication-total | The total number of connections with failed authentication | client-id | | | | | +| producer-metrics | failed-reauthentication-rate | The number of failed re-authentication of connections per second | client-id | | | | | +| producer-metrics | failed-reauthentication-total | The total number of failed re-authentication of connections | client-id | | | | | +| producer-metrics | incoming-byte-rate | The number of bytes read off all sockets per second | client-id | | | | | +| producer-metrics | incoming-byte-total | The total number of bytes read off all sockets | client-id | | | | | +| producer-metrics | io-ratio | The fraction of time the I/O thread spent doing I/O | client-id | | | | | +| producer-metrics | io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | client-id | | | | | +| producer-metrics | io-wait-ratio | The fraction of time the I/O thread spent waiting | client-id | | | | | +| producer-metrics | io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | client-id | | | | | +| producer-metrics | io-waittime-total | The total time the I/O thread spent waiting | client-id | | | | | +| producer-metrics | iotime-total | The total time the I/O thread spent doing I/O | client-id | | | | | +| producer-metrics | metadata-age | The age in seconds of the current producer metadata being used. | client-id | | | | | +| producer-metrics | network-io-rate | The number of network operations (reads or writes) on all connections per second | client-id | | | | | +| producer-metrics | network-io-total | The total number of network operations (reads or writes) on all connections | client-id | | | | | +| producer-metrics | outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | client-id | messaging.kafka.producer.outgoing-bytes.rate | The average number of outgoing bytes sent per second to all servers. | by/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-metrics | outgoing-byte-total | The total number of outgoing bytes sent to all servers | client-id | | | | | +| producer-metrics | produce-throttle-time-avg | The average time in ms a request was throttled by a broker | client-id | | | | | +| producer-metrics | produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | client-id | | | | | +| producer-metrics | reauthentication-latency-avg | The average latency observed due to re-authentication | client-id | | | | | +| producer-metrics | reauthentication-latency-max | The max latency observed due to re-authentication | client-id | | | | | +| producer-metrics | record-error-rate | The average per-second number of record sends that resulted in errors | client-id | | | | | +| producer-metrics | record-error-total | The total number of record sends that resulted in errors | client-id | | | | | +| producer-metrics | record-queue-time-avg | The average time in ms record batches spent in the send buffer. | client-id | | | | | +| producer-metrics | record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | client-id | | | | | +| producer-metrics | record-retry-rate | The average per-second number of retried record sends | client-id | | | | | +| producer-metrics | record-retry-total | The total number of retried record sends | client-id | | | | | +| producer-metrics | record-send-rate | The average number of records sent per second. | client-id | | | | | +| producer-metrics | record-send-total | The total number of records sent. | client-id | | | | | +| producer-metrics | record-size-avg | The average record size | client-id | | | | | +| producer-metrics | record-size-max | The maximum record size | client-id | | | | | +| producer-metrics | records-per-request-avg | The average number of records per request. | client-id | | | | | +| producer-metrics | request-latency-avg | The average request latency in ms | client-id | | | | | +| producer-metrics | request-latency-max | The maximum request latency in ms | client-id | | | | | +| producer-metrics | request-rate | The number of requests sent per second | client-id | | | | | +| producer-metrics | request-size-avg | The average size of requests sent. | client-id | | | | | +| producer-metrics | request-size-max | The maximum size of any request sent. | client-id | | | | | +| producer-metrics | request-total | The total number of requests sent | client-id | | | | | +| producer-metrics | requests-in-flight | The current number of in-flight requests awaiting a response. | client-id | | | | | +| producer-metrics | response-rate | The number of responses received per second | client-id | messaging.kafka.producer.responses.rate | The average number of responses received per second. | {responses}/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-metrics | response-total | The total number of responses received | client-id | | | | | +| producer-metrics | select-rate | The number of times the I/O layer checked for new I/O to perform per second | client-id | | | | | +| producer-metrics | select-total | The total number of times the I/O layer checked for new I/O to perform | client-id | | | | | +| producer-metrics | successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | client-id | | | | | +| producer-metrics | successful-authentication-rate | The number of connections with successful authentication per second | client-id | | | | | +| producer-metrics | successful-authentication-total | The total number of connections with successful authentication | client-id | | | | | +| producer-metrics | successful-reauthentication-rate | The number of successful re-authentication of connections per second | client-id | | | | | +| producer-metrics | successful-reauthentication-total | The total number of successful re-authentication of connections | client-id | | | | | +| producer-metrics | waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | client-id | | | | | +| producer-node-metrics | incoming-byte-rate | The number of incoming bytes per second | client-id,node-id | | | | | +| producer-node-metrics | incoming-byte-total | The total number of incoming bytes | client-id,node-id | | | | | +| producer-node-metrics | outgoing-byte-rate | The number of outgoing bytes per second | client-id,node-id | | | | | +| producer-node-metrics | outgoing-byte-total | The total number of outgoing bytes | client-id,node-id | | | | | +| producer-node-metrics | request-latency-avg | | client-id,node-id | | | | | +| producer-node-metrics | request-latency-max | | client-id,node-id | | | | | +| producer-node-metrics | request-rate | The number of requests sent per second | client-id,node-id | | | | | +| producer-node-metrics | request-size-avg | The average size of requests sent. | client-id,node-id | | | | | +| producer-node-metrics | request-size-max | The maximum size of any request sent. | client-id,node-id | | | | | +| producer-node-metrics | request-total | The total number of requests sent | client-id,node-id | | | | | +| producer-node-metrics | response-rate | The number of responses received per second | client-id,node-id | | | | | +| producer-node-metrics | response-total | The total number of responses received | client-id,node-id | | | | | +| producer-topic-metrics | byte-rate | The average number of bytes sent per second for a topic. | client-id,topic | messaging.kafka.producer.bytes.rate | The average number of bytes sent per second for a specific topic. | by/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-topic-metrics | byte-total | The total number of bytes sent for a topic. | client-id,topic | | | | | +| producer-topic-metrics | compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | client-id,topic | messaging.kafka.producer.compression-ratio | The average compression ratio of record batches for a specific topic. | {compression} | DOUBLE_OBSERVABLE_GAUGE | +| producer-topic-metrics | record-error-rate | The average per-second number of record sends that resulted in errors for a topic | client-id,topic | messaging.kafka.producer.record-error.rate | The average per-second number of record sends that resulted in errors for a specific topic. | {errors}/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-topic-metrics | record-error-total | The total number of record sends that resulted in errors for a topic | client-id,topic | | | | | +| producer-topic-metrics | record-retry-rate | The average per-second number of retried record sends for a topic | client-id,topic | messaging.kafka.producer.record-retry.rate | The average per-second number of retried record sends for a specific topic. | {retries}/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-topic-metrics | record-retry-total | The total number of retried record sends for a topic | client-id,topic | | | | | +| producer-topic-metrics | record-send-rate | The average number of records sent per second for a topic. | client-id,topic | messaging.kafka.producer.record-sent.rate | The average number of records sent per second for a specific topic. | {records_sent}/s | DOUBLE_OBSERVABLE_GAUGE | +| producer-topic-metrics | record-send-total | The total number of records sent for a topic. | client-id,topic | | | | | \ No newline at end of file diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts index b8dc4adc702b..28b72b16f5a0 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts @@ -10,4 +10,9 @@ dependencies { implementation(project(":instrumentation:kafka:kafka-clients:kafka-clients-common:library")) implementation("org.testcontainers:kafka") + + runtimeOnly("org.apache.kafka:kafka_2.13:2.8.1") + implementation("com.salesforce.kafka.test:kafka-junit5:3.2.3") { + exclude(module="zookeeper") // conflicts with zookeeper from kafka_2.x + } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java new file mode 100644 index 000000000000..318dabdc000e --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -0,0 +1,98 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import com.salesforce.kafka.test.junit5.SharedKafkaTestResource; +import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension; +import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension; +import java.nio.charset.StandardCharsets; +import java.time.Instant; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Random; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.ByteArrayDeserializer; +import org.apache.kafka.common.serialization.ByteArraySerializer; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +abstract class OpenTelemetryKafkaMetricsTest { + + private static final List TOPICS = Arrays.asList("foo", "bar", "baz", "qux"); + private static final Random RANDOM = new Random(); + + @RegisterExtension + static final SharedKafkaTestResource kafka = new SharedKafkaTestResource(); + + @RegisterExtension + static final InstrumentationExtension testing = LibraryInstrumentationExtension.create(); + + @Test + void observeMetrics() { + OpenTelemetryKafkaMetrics.setOpenTelemetry(testing.getOpenTelemetry()); + + produceRecords(); + consumeRecords(); + + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.outgoing-bytes.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.responses.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.bytes.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.compression-ratio", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-error.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-retry.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-sent.rate", unused -> {}); + testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.consumer.lag", unused -> {}); + + // Print mapping table + OpenTelemetryKafkaMetrics.printMappingTable(); + } + + void produceRecords() { + Map config = new HashMap<>(); + // Register OpenTelemetryKafkaMetrics as reporter + config.put( + ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); + config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); + config.put(ProducerConfig.CLIENT_ID_CONFIG, "sample-client-id"); + config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); + config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); + config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip"); + + try (KafkaProducer producer = new KafkaProducer<>(config)) { + for (int i = 0; i < 100; i++) { + producer.send( + new ProducerRecord<>( + TOPICS.get(RANDOM.nextInt(TOPICS.size())), + 0, + System.currentTimeMillis(), + "key".getBytes(StandardCharsets.UTF_8), + "value".getBytes(StandardCharsets.UTF_8))); + } + } + } + + void consumeRecords() { + Map config = new HashMap<>(); + // Register OpenTelemetryKafkaMetrics as reporter + config.put( + ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); + config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); + config.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-group"); + config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); + config.put( + ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); + config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2); + + try (KafkaConsumer consumer = new KafkaConsumer<>(config)) { + consumer.subscribe(TOPICS); + Instant stopTime = Instant.now().plusSeconds(10); + while (Instant.now().isBefore(stopTime)) { + consumer.poll(1000); + } + } + } +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java new file mode 100644 index 000000000000..03df474b6c88 --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java @@ -0,0 +1,4 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +public class MetricsTest extends OpenTelemetryKafkaMetricsTest { +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java new file mode 100644 index 000000000000..b283e8716d0b --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java @@ -0,0 +1,100 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import java.util.Set; +import org.apache.kafka.common.MetricName; +import org.apache.kafka.common.metrics.KafkaMetric; +import javax.annotation.Nullable; + +/** + * A value class collecting the identifying fields of a kafka {@link MetricName}. + * + *

Note: {@link #description} is included in {@link #toString()} but omitted from {@link + * #equals(Object)} and {@link #hashCode()}. + */ +class KafkaMetricId { + + private final String name; + private final String group; + private final String description; + private final Set tagKeys; + + private KafkaMetricId(String name, String group, String description, Set tagKeys) { + this.name = name; + this.group = group; + this.description = description; + this.tagKeys = tagKeys; + } + + static KafkaMetricId create(KafkaMetric kafkaMetric) { + return new KafkaMetricId( + kafkaMetric.metricName().name(), + kafkaMetric.metricName().group(), + kafkaMetric.metricName().description(), + kafkaMetric.metricName().tags().keySet()); + } + + static KafkaMetricId create(String name, String group, Set tagKeys) { + return new KafkaMetricId(name, group, "", tagKeys); + } + + String getName() { + return name; + } + + String getGroup() { + return group; + } + + String getDescription() { + return description; + } + + Set getTagKeys() { + return tagKeys; + } + + @Override + public String toString() { + return "KafkaMetricId{" + + "name=" + + name + + ", " + + "group=" + + group + + ", " + + "description=" + + description + + ", " + + "tagKeys=" + + tagKeys + + "}"; + } + + @Override + public boolean equals(@Nullable Object o) { + if (o == this) { + return true; + } + if (o instanceof KafkaMetricId) { + KafkaMetricId that = (KafkaMetricId) o; + // Omit description from equality + return this.name.equals(that.getName()) + && this.group.equals(that.getGroup()) + && this.tagKeys.equals(that.getTagKeys()); + } + return false; + } + + @Override + public int hashCode() { + // Omit description from hashcode + int hash = 1; + hash *= 1000003; + hash ^= name.hashCode(); + hash *= 1000003; + hash ^= group.hashCode(); + hash *= 1000003; + hash ^= tagKeys.hashCode(); + return hash; + } +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java new file mode 100644 index 000000000000..a918b948c27c --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -0,0 +1,105 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */ +class KafkaMetricRegistry { + + private static final String unitBytesPerSecond = "by/s"; + + private static final Map registry = new HashMap<>(); + + static { + registerProducerMetrics(); + registerProducerTopicMetrics(); + registerConsumerMetrics(); + } + + private static void registerProducerMetrics() { + String group = "producer-metrics"; + Set tagKeys = Collections.singleton("client-id"); + + registry.put( + KafkaMetricId.create("outgoing-byte-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.outgoing-bytes.rate", + "The average number of outgoing bytes sent per second to all servers.", + unitBytesPerSecond)); + registry.put( + KafkaMetricId.create("response-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.responses.rate", + "The average number of responses received per second.", + "{responses}/s")); + } + + private static void registerProducerTopicMetrics() { + String group = "producer-topic-metrics"; + Set tagKeys = new HashSet<>(Arrays.asList("client-id", "topic")); + + registry.put( + KafkaMetricId.create("byte-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.bytes.rate", + "The average number of bytes sent per second for a specific topic.", + unitBytesPerSecond)); + registry.put( + KafkaMetricId.create("compression-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.compression-ratio", + "The average compression ratio of record batches for a specific topic.", + "{compression}")); + registry.put( + KafkaMetricId.create("record-error-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.record-error.rate", + "The average per-second number of record sends that resulted in errors for a specific topic.", + "{errors}/s")); + registry.put( + KafkaMetricId.create("record-retry-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.record-retry.rate", + "The average per-second number of retried record sends for a specific topic.", + "{retries}/s")); + registry.put( + KafkaMetricId.create("record-send-rate", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.producer.record-sent.rate", + "The average number of records sent per second for a specific topic.", + "{records_sent}/s")); + } + + private static void registerConsumerMetrics() { + String group = "consumer-fetch-manager-metrics"; + Set tagKeys = new HashSet<>(Arrays.asList("client-id", "topic", "partition")); + + registry.put( + KafkaMetricId.create("records-lag", group, tagKeys), + MetricDescriptor.createDoubleGauge( + "messaging.kafka.consumer.lag", + "Current approximate lag of consumer group at partition of topic.", + "{lag}")); + } + + /** + * Returns the description of the OpenTelemetry metric definition for the kafka metric, or {@code + * null} if no mapping exists. + */ + @Nullable + static MetricDescriptor getRegisteredInstrument(KafkaMetricId kafkaMetricId) { + return registry.get(kafkaMetricId); + } + + // Visible for testing + static Set getMetricDescriptors() { + return new HashSet<>(registry.values()); + } + + private KafkaMetricRegistry() {} +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java new file mode 100644 index 000000000000..10be5bbba7ad --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java @@ -0,0 +1,23 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import com.google.auto.value.AutoValue; + +/** A description of an OpenTelemetry metric instrument. */ +@AutoValue +abstract class MetricDescriptor { + + static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE = "DOUBLE_OBSERVABLE_GAUGE"; + + abstract String getName(); + + abstract String getDescription(); + + abstract String getUnit(); + + abstract String getInstrumentType(); + + static MetricDescriptor createDoubleGauge(String name, String description, String unit) { + return new AutoValue_MetricDescriptor( + name, description, unit, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + } +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java new file mode 100644 index 000000000000..9af51ae8e5f8 --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -0,0 +1,201 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import static java.lang.System.lineSeparator; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.toList; + +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.common.Attributes; +import io.opentelemetry.api.common.AttributesBuilder; +import io.opentelemetry.api.metrics.Meter; +import java.util.Comparator; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.logging.Level; +import java.util.logging.Logger; +import javax.annotation.Nullable; +import org.apache.kafka.common.metrics.KafkaMetric; +import org.apache.kafka.common.metrics.MetricsReporter; + +/** + * A {@link MetricsReporter} which bridges Kafka metrics to OpenTelemetry metrics. + * + *

To use, configure OpenTelemetry instance via {@link #setOpenTelemetry(OpenTelemetry)}, and + * include a reference to this class in kafka producer or consumer configuration, i.e.: + * + * 

{@code
+ * //    Map config = new HashMap<>();
+ * //    // Register OpenTelemetryKafkaMetrics as reporter
+ * //    config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName());
+ * //    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString());
+ * //    ...
+ * //    try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
+ * }
+ */ +public class OpenTelemetryKafkaMetrics implements MetricsReporter { + + private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName()); + private static final Set seenMetrics = ConcurrentHashMap.newKeySet(); + + @Nullable private static volatile Meter meter; + + private static final Map instrumentMap = + new ConcurrentHashMap<>(); + + /** + * Setup OpenTelemetry. This should be called as early in the application lifecycle as possible. + * Kafka metrics that are observed before this is called may not be bridged. + */ + public static void setOpenTelemetry(OpenTelemetry openTelemetry) { + meter = openTelemetry.getMeter("io.opentelemetry.kafka-clients"); + } + + /** + * Reset for test by reseting the {@link #meter} to {@code null} and closing all registered + * instruments. + */ + static void resetForTest() { + meter = null; + for(Iterator> it = instrumentMap.entrySet().iterator(); it.hasNext(); ) { + try { + it.next().getValue().close(); + it.remove(); + } catch (Exception e) { + throw new IllegalStateException("Error occurred closing instrument", e); + } + } + } + + /** + * Print a table mapping kafka metrics to equivalent OpenTelemetry metrics, in markdown format. + */ + static void printMappingTable() { + StringBuilder sb = new StringBuilder(); + // Append table headers + sb.append( + "| Kafka Group | Kafka Name | Kafka Description | Attribute Keys | Instrument Name | Instrument Description | Instrument Unit | Instrument Type |") + .append(lineSeparator()) + .append( + "|-------------|------------|-------------------|----------------|-----------------|------------------------|-----------------|-----------------|") + .append(lineSeparator()); + Map> kafkaMetricsByGroup = + seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup)); + // Iterate through groups in alpha order + for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { + List kafkaMetricIds = + kafkaMetricsByGroup.get(group).stream() + .sorted(Comparator.comparing(KafkaMetricId::getName)) + .collect(toList()); + // Iterate through metrics in alpha order by name + for (KafkaMetricId kafkaMetricId : kafkaMetricIds) { + Optional descriptor = + Optional.ofNullable(KafkaMetricRegistry.getRegisteredInstrument(kafkaMetricId)); + // Append table row + sb.append( + String.format( + "| %s | %s | %s | %s | %s | %s | %s | %s |%n", + group, + kafkaMetricId.getName(), + kafkaMetricId.getDescription(), + String.join(",", kafkaMetricId.getTagKeys()), + descriptor.map(MetricDescriptor::getName).orElse(""), + descriptor.map(MetricDescriptor::getDescription).orElse(""), + descriptor.map(MetricDescriptor::getUnit).orElse(""), + descriptor + .map(MetricDescriptor::getInstrumentType) + .orElse(""))); + } + } + logger.log(Level.INFO, "Mapping table" + System.lineSeparator() + sb); + } + + @Override + public void init(List metrics) { + metrics.forEach(this::metricChange); + } + + @Override + public void metricChange(KafkaMetric metric) { + KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); + seenMetrics.add(KafkaMetricId.create(metric)); + Meter currentMeter = meter; + if (currentMeter == null) { + logger.log(Level.FINEST, "Metric changed but meter not set: " + kafkaMetricId); + return; + } + + MetricDescriptor metricDescriptor = KafkaMetricRegistry.getRegisteredInstrument(kafkaMetricId); + if (metricDescriptor == null) { + logger.log( + Level.FINEST, + "Metric changed but did not match any metrics from registry: " + kafkaMetricId); + return; + } + + AttributesBuilder attributesBuilder = Attributes.builder(); + metric.metricName().tags().forEach(attributesBuilder::put); + RegisteredInstrument registeredInstrument = + RegisteredInstrument.create(kafkaMetricId, attributesBuilder.build()); + + instrumentMap.compute( + registeredInstrument, + (registeredInstrument1, autoCloseable) -> { + if (autoCloseable != null) { + logger.log(Level.FINEST, "Replacing instrument " + registeredInstrument1); + try { + autoCloseable.close(); + } catch (Exception e) { + logger.log(Level.WARNING, "An error occurred closing instrument", e); + } + } else { + logger.log(Level.FINEST, "Adding instrument " + registeredInstrument1); + } + return createObservable(currentMeter, registeredInstrument1, metricDescriptor, metric); + }); + } + + private static AutoCloseable createObservable( + Meter meter, + RegisteredInstrument registeredInstrument, + MetricDescriptor metricDescriptor, + KafkaMetric kafkaMetric) { + if (metricDescriptor.getInstrumentType().equals(MetricDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE)) { + return meter + .gaugeBuilder(metricDescriptor.getName()) + .setDescription(metricDescriptor.getDescription()) + .setUnit(metricDescriptor.getUnit()) + .buildWithCallback( + observableMeasurement -> + observableMeasurement.record( + kafkaMetric.value(), registeredInstrument.getAttributes())); + } + // TODO: add support for other instrument types and value types as needed for new instruments + // registered in KafkaMetricRegistry. + // This should not happen. + throw new IllegalStateException("Unsupported metric descriptor: " + metricDescriptor); + } + + private static Attributes toAttributes(KafkaMetric kafkaMetric) { + AttributesBuilder attributesBuilder = Attributes.builder(); + kafkaMetric.metricName().tags().forEach(attributesBuilder::put); + return attributesBuilder.build(); + } + + @Override + public void metricRemoval(KafkaMetric metric) { + KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); + seenMetrics.add(kafkaMetricId); + logger.log(Level.FINEST, "Metric removed: " + kafkaMetricId); + instrumentMap.remove(RegisteredInstrument.create(kafkaMetricId, toAttributes(metric))); + } + + @Override + public void close() {} + + @Override + public void configure(Map configs) {} +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java new file mode 100644 index 000000000000..9cf24c0a3359 --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java @@ -0,0 +1,16 @@ +package io.opentelemetry.instrumentation.kafkaclients; + +import com.google.auto.value.AutoValue; +import io.opentelemetry.api.common.Attributes; + +@AutoValue +abstract class RegisteredInstrument { + + abstract KafkaMetricId getKafkaMetricId(); + + abstract Attributes getAttributes(); + + static RegisteredInstrument create(KafkaMetricId kafkaMetricId, Attributes attributes) { + return new AutoValue_RegisteredInstrument(kafkaMetricId, attributes); + } +} From 10b67e33990447d6eba6c06dadd9b30f5098ad66 Mon Sep 17 00:00:00 2001 From: jack-berg Date: Sat, 4 Jun 2022 14:24:40 -0500 Subject: [PATCH 02/17] Spotless --- .../testing/build.gradle.kts | 2 +- .../OpenTelemetryKafkaMetricsTest.java | 42 ++++++++++++++----- .../kafkaclients/MetricsTest.java | 8 +++- .../kafkaclients/KafkaMetricId.java | 7 +++- .../kafkaclients/KafkaMetricRegistry.java | 5 +++ .../kafkaclients/MetricDescriptor.java | 5 +++ .../OpenTelemetryKafkaMetrics.java | 17 +++++--- .../kafkaclients/RegisteredInstrument.java | 5 +++ 8 files changed, 72 insertions(+), 19 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts index 28b72b16f5a0..cab63599bcba 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts @@ -13,6 +13,6 @@ dependencies { runtimeOnly("org.apache.kafka:kafka_2.13:2.8.1") implementation("com.salesforce.kafka.test:kafka-junit5:3.2.3") { - exclude(module="zookeeper") // conflicts with zookeeper from kafka_2.x + exclude(module = "zookeeper") // conflicts with zookeeper from kafka_2.x } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index 318dabdc000e..dbd87631becd 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -1,3 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import com.salesforce.kafka.test.junit5.SharedKafkaTestResource; @@ -25,8 +30,7 @@ abstract class OpenTelemetryKafkaMetricsTest { private static final List TOPICS = Arrays.asList("foo", "bar", "baz", "qux"); private static final Random RANDOM = new Random(); - @RegisterExtension - static final SharedKafkaTestResource kafka = new SharedKafkaTestResource(); + @RegisterExtension static final SharedKafkaTestResource kafka = new SharedKafkaTestResource(); @RegisterExtension static final InstrumentationExtension testing = LibraryInstrumentationExtension.create(); @@ -38,14 +42,32 @@ void observeMetrics() { produceRecords(); consumeRecords(); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.outgoing-bytes.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.responses.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.bytes.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.compression-ratio", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-error.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-retry.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.producer.record-sent.rate", unused -> {}); - testing.waitAndAssertMetrics("io.opentelemetry.kafka-clients", "messaging.kafka.consumer.lag", unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", + "messaging.kafka.producer.outgoing-bytes.rate", + unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", "messaging.kafka.producer.responses.rate", unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", "messaging.kafka.producer.bytes.rate", unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", + "messaging.kafka.producer.compression-ratio", + unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", + "messaging.kafka.producer.record-error.rate", + unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", + "messaging.kafka.producer.record-retry.rate", + unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", + "messaging.kafka.producer.record-sent.rate", + unused -> {}); + testing.waitAndAssertMetrics( + "io.opentelemetry.kafka-clients", "messaging.kafka.consumer.lag", unused -> {}); // Print mapping table OpenTelemetryKafkaMetrics.printMappingTable(); diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java index 03df474b6c88..0c3bc151f4e3 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java @@ -1,4 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; -public class MetricsTest extends OpenTelemetryKafkaMetricsTest { -} +public class MetricsTest extends OpenTelemetryKafkaMetricsTest {} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java index b283e8716d0b..598d821d9db4 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java @@ -1,9 +1,14 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import java.util.Set; +import javax.annotation.Nullable; import org.apache.kafka.common.MetricName; import org.apache.kafka.common.metrics.KafkaMetric; -import javax.annotation.Nullable; /** * A value class collecting the identifying fields of a kafka {@link MetricName}. diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index a918b948c27c..385fe26a200e 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -1,3 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import java.util.Arrays; diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java index 10be5bbba7ad..44ca7b8eec5a 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java @@ -1,3 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import com.google.auto.value.AutoValue; diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 9af51ae8e5f8..c657c0114605 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -1,3 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import static java.lang.System.lineSeparator; @@ -60,7 +65,9 @@ public static void setOpenTelemetry(OpenTelemetry openTelemetry) { */ static void resetForTest() { meter = null; - for(Iterator> it = instrumentMap.entrySet().iterator(); it.hasNext(); ) { + for (Iterator> it = + instrumentMap.entrySet().iterator(); + it.hasNext(); ) { try { it.next().getValue().close(); it.remove(); @@ -105,9 +112,7 @@ static void printMappingTable() { descriptor.map(MetricDescriptor::getName).orElse(""), descriptor.map(MetricDescriptor::getDescription).orElse(""), descriptor.map(MetricDescriptor::getUnit).orElse(""), - descriptor - .map(MetricDescriptor::getInstrumentType) - .orElse(""))); + descriptor.map(MetricDescriptor::getInstrumentType).orElse(""))); } } logger.log(Level.INFO, "Mapping table" + System.lineSeparator() + sb); @@ -163,7 +168,9 @@ private static AutoCloseable createObservable( RegisteredInstrument registeredInstrument, MetricDescriptor metricDescriptor, KafkaMetric kafkaMetric) { - if (metricDescriptor.getInstrumentType().equals(MetricDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE)) { + if (metricDescriptor + .getInstrumentType() + .equals(MetricDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE)) { return meter .gaugeBuilder(metricDescriptor.getName()) .setDescription(metricDescriptor.getDescription()) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java index 9cf24c0a3359..341ed5bba7eb 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java @@ -1,3 +1,8 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + package io.opentelemetry.instrumentation.kafkaclients; import com.google.auto.value.AutoValue; From e49f04b52499082340e7d6b3b1c3a48e325e5ba7 Mon Sep 17 00:00:00 2001 From: jack-berg Date: Wed, 8 Jun 2022 16:23:36 -0500 Subject: [PATCH 03/17] Refactor to general purpose bridge between kafka client metrics and opentelemetry metrics --- instrumentation/kafka/kafka-clients/README.md | 408 +++++++++--------- .../OpenTelemetryKafkaMetricsTest.java | 228 ++++++++-- .../kafkaclients/InstrumentDescriptor.java | 39 ++ .../kafkaclients/KafkaMetricId.java | 108 +---- .../kafkaclients/KafkaMetricRegistry.java | 139 +++--- .../kafkaclients/MetricDescriptor.java | 28 -- .../OpenTelemetryKafkaMetrics.java | 83 ++-- 7 files changed, 572 insertions(+), 461 deletions(-) create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java delete mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java diff --git a/instrumentation/kafka/kafka-clients/README.md b/instrumentation/kafka/kafka-clients/README.md index f027e0c895f8..8ed4ea06f18e 100644 --- a/instrumentation/kafka/kafka-clients/README.md +++ b/instrumentation/kafka/kafka-clients/README.md @@ -27,207 +27,207 @@ OpenTelemetry metric each maps to (if available). Empty values in the Instrument Description, etc column indicates there is no registered mapping for the metric and data is NOT collected. -| Kafka Group | Kafka Name | Kafka Description | Attribute Keys | Instrument Name | Instrument Description | Instrument Unit | Instrument Type | -|-------------|------------|-------------------|----------------|-----------------|------------------------|-----------------|-----------------| -| app-info | commit-id | Metric indicating commit-id | client-id | | | | | -| app-info | start-time-ms | Metric indicating start-time-ms | client-id | | | | | -| app-info | version | Metric indicating version | client-id | | | | | -| consumer-coordinator-metrics | assigned-partitions | The number of partitions currently assigned to this consumer | client-id | | | | | -| consumer-coordinator-metrics | commit-latency-avg | The average time taken for a commit request | client-id | | | | | -| consumer-coordinator-metrics | commit-latency-max | The max time taken for a commit request | client-id | | | | | -| consumer-coordinator-metrics | commit-rate | The number of commit calls per second | client-id | | | | | -| consumer-coordinator-metrics | commit-total | The total number of commit calls | client-id | | | | | -| consumer-coordinator-metrics | failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | client-id | | | | | -| consumer-coordinator-metrics | failed-rebalance-total | The total number of failed rebalance events | client-id | | | | | -| consumer-coordinator-metrics | heartbeat-rate | The number of heartbeats per second | client-id | | | | | -| consumer-coordinator-metrics | heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | client-id | | | | | -| consumer-coordinator-metrics | heartbeat-total | The total number of heartbeats | client-id | | | | | -| consumer-coordinator-metrics | join-rate | The number of group joins per second | client-id | | | | | -| consumer-coordinator-metrics | join-time-avg | The average time taken for a group rejoin | client-id | | | | | -| consumer-coordinator-metrics | join-time-max | The max time taken for a group rejoin | client-id | | | | | -| consumer-coordinator-metrics | join-total | The total number of group joins | client-id | | | | | -| consumer-coordinator-metrics | last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | client-id | | | | | -| consumer-coordinator-metrics | last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | client-id | | | | | -| consumer-coordinator-metrics | partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | client-id | | | | | -| consumer-coordinator-metrics | rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | client-id | | | | | -| consumer-coordinator-metrics | rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | client-id | | | | | -| consumer-coordinator-metrics | rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | client-id | | | | | -| consumer-coordinator-metrics | rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | client-id | | | | | -| consumer-coordinator-metrics | rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | client-id | | | | | -| consumer-coordinator-metrics | sync-rate | The number of group syncs per second | client-id | | | | | -| consumer-coordinator-metrics | sync-time-avg | The average time taken for a group sync | client-id | | | | | -| consumer-coordinator-metrics | sync-time-max | The max time taken for a group sync | client-id | | | | | -| consumer-coordinator-metrics | sync-total | The total number of group syncs | client-id | | | | | -| consumer-fetch-manager-metrics | bytes-consumed-rate | The average number of bytes consumed per second | client-id | | | | | -| consumer-fetch-manager-metrics | bytes-consumed-rate | The average number of bytes consumed per second for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | bytes-consumed-total | The total number of bytes consumed for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | bytes-consumed-total | The total number of bytes consumed | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-latency-avg | The average time taken for a fetch request. | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-latency-max | The max time taken for any fetch request. | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-rate | The number of fetch requests per second. | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-size-avg | The average number of bytes fetched per request | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-size-avg | The average number of bytes fetched per request for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | fetch-size-max | The maximum number of bytes fetched per request for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | fetch-size-max | The maximum number of bytes fetched per request | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-throttle-time-avg | The average throttle time in ms | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-throttle-time-max | The maximum throttle time in ms | client-id | | | | | -| consumer-fetch-manager-metrics | fetch-total | The total number of fetch requests. | client-id | | | | | -| consumer-fetch-manager-metrics | preferred-read-replica | The current read replica for the partition, or -1 if reading from leader | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-consumed-rate | The average number of records consumed per second | client-id | | | | | -| consumer-fetch-manager-metrics | records-consumed-rate | The average number of records consumed per second for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | records-consumed-total | The total number of records consumed for a topic | client-id,topic | | | | | -| consumer-fetch-manager-metrics | records-consumed-total | The total number of records consumed | client-id | | | | | -| consumer-fetch-manager-metrics | records-lag | The latest lag of the partition | client-id,topic,partition | messaging.kafka.consumer.lag | Current approximate lag of consumer group at partition of topic. | {lag} | DOUBLE_OBSERVABLE_GAUGE | -| consumer-fetch-manager-metrics | records-lag-avg | The average lag of the partition | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-lag-max | The maximum lag in terms of number of records for any partition in this window | client-id | | | | | -| consumer-fetch-manager-metrics | records-lag-max | The max lag of the partition | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-lead | The latest lead of the partition | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-lead-avg | The average lead of the partition | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-lead-min | The minimum lead in terms of number of records for any partition in this window | client-id | | | | | -| consumer-fetch-manager-metrics | records-lead-min | The min lead of the partition | client-id,topic,partition | | | | | -| consumer-fetch-manager-metrics | records-per-request-avg | The average number of records in each request | client-id | | | | | -| consumer-fetch-manager-metrics | records-per-request-avg | The average number of records in each request for a topic | client-id,topic | | | | | -| consumer-metrics | connection-close-rate | The number of connections closed per second | client-id | | | | | -| consumer-metrics | connection-close-total | The total number of connections closed | client-id | | | | | -| consumer-metrics | connection-count | The current number of active connections. | client-id | | | | | -| consumer-metrics | connection-creation-rate | The number of new connections established per second | client-id | | | | | -| consumer-metrics | connection-creation-total | The total number of new connections established | client-id | | | | | -| consumer-metrics | failed-authentication-rate | The number of connections with failed authentication per second | client-id | | | | | -| consumer-metrics | failed-authentication-total | The total number of connections with failed authentication | client-id | | | | | -| consumer-metrics | failed-reauthentication-rate | The number of failed re-authentication of connections per second | client-id | | | | | -| consumer-metrics | failed-reauthentication-total | The total number of failed re-authentication of connections | client-id | | | | | -| consumer-metrics | incoming-byte-rate | The number of bytes read off all sockets per second | client-id | | | | | -| consumer-metrics | incoming-byte-total | The total number of bytes read off all sockets | client-id | | | | | -| consumer-metrics | io-ratio | The fraction of time the I/O thread spent doing I/O | client-id | | | | | -| consumer-metrics | io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | client-id | | | | | -| consumer-metrics | io-wait-ratio | The fraction of time the I/O thread spent waiting | client-id | | | | | -| consumer-metrics | io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | client-id | | | | | -| consumer-metrics | io-waittime-total | The total time the I/O thread spent waiting | client-id | | | | | -| consumer-metrics | iotime-total | The total time the I/O thread spent doing I/O | client-id | | | | | -| consumer-metrics | last-poll-seconds-ago | The number of seconds since the last poll() invocation. | client-id | | | | | -| consumer-metrics | network-io-rate | The number of network operations (reads or writes) on all connections per second | client-id | | | | | -| consumer-metrics | network-io-total | The total number of network operations (reads or writes) on all connections | client-id | | | | | -| consumer-metrics | outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | client-id | | | | | -| consumer-metrics | outgoing-byte-total | The total number of outgoing bytes sent to all servers | client-id | | | | | -| consumer-metrics | poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | client-id | | | | | -| consumer-metrics | reauthentication-latency-avg | The average latency observed due to re-authentication | client-id | | | | | -| consumer-metrics | reauthentication-latency-max | The max latency observed due to re-authentication | client-id | | | | | -| consumer-metrics | request-rate | The number of requests sent per second | client-id | | | | | -| consumer-metrics | request-size-avg | The average size of requests sent. | client-id | | | | | -| consumer-metrics | request-size-max | The maximum size of any request sent. | client-id | | | | | -| consumer-metrics | request-total | The total number of requests sent | client-id | | | | | -| consumer-metrics | response-rate | The number of responses received per second | client-id | | | | | -| consumer-metrics | response-total | The total number of responses received | client-id | | | | | -| consumer-metrics | select-rate | The number of times the I/O layer checked for new I/O to perform per second | client-id | | | | | -| consumer-metrics | select-total | The total number of times the I/O layer checked for new I/O to perform | client-id | | | | | -| consumer-metrics | successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | client-id | | | | | -| consumer-metrics | successful-authentication-rate | The number of connections with successful authentication per second | client-id | | | | | -| consumer-metrics | successful-authentication-total | The total number of connections with successful authentication | client-id | | | | | -| consumer-metrics | successful-reauthentication-rate | The number of successful re-authentication of connections per second | client-id | | | | | -| consumer-metrics | successful-reauthentication-total | The total number of successful re-authentication of connections | client-id | | | | | -| consumer-metrics | time-between-poll-avg | The average delay between invocations of poll(). | client-id | | | | | -| consumer-metrics | time-between-poll-max | The max delay between invocations of poll(). | client-id | | | | | -| consumer-node-metrics | incoming-byte-rate | The number of incoming bytes per second | client-id,node-id | | | | | -| consumer-node-metrics | incoming-byte-total | The total number of incoming bytes | client-id,node-id | | | | | -| consumer-node-metrics | outgoing-byte-rate | The number of outgoing bytes per second | client-id,node-id | | | | | -| consumer-node-metrics | outgoing-byte-total | The total number of outgoing bytes | client-id,node-id | | | | | -| consumer-node-metrics | request-latency-avg | | client-id,node-id | | | | | -| consumer-node-metrics | request-latency-max | | client-id,node-id | | | | | -| consumer-node-metrics | request-rate | The number of requests sent per second | client-id,node-id | | | | | -| consumer-node-metrics | request-size-avg | The average size of requests sent. | client-id,node-id | | | | | -| consumer-node-metrics | request-size-max | The maximum size of any request sent. | client-id,node-id | | | | | -| consumer-node-metrics | request-total | The total number of requests sent | client-id,node-id | | | | | -| consumer-node-metrics | response-rate | The number of responses received per second | client-id,node-id | | | | | -| consumer-node-metrics | response-total | The total number of responses received | client-id,node-id | | | | | -| kafka-metrics-count | count | total number of registered metrics | client-id | | | | | -| producer-metrics | batch-size-avg | The average number of bytes sent per partition per-request. | client-id | | | | | -| producer-metrics | batch-size-max | The max number of bytes sent per partition per-request. | client-id | | | | | -| producer-metrics | batch-split-rate | The average number of batch splits per second | client-id | | | | | -| producer-metrics | batch-split-total | The total number of batch splits | client-id | | | | | -| producer-metrics | buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | client-id | | | | | -| producer-metrics | buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | client-id | | | | | -| producer-metrics | buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | client-id | | | | | -| producer-metrics | buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | client-id | | | | | -| producer-metrics | bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | client-id | | | | | -| producer-metrics | bufferpool-wait-time-total | The total time an appender waits for space allocation. | client-id | | | | | -| producer-metrics | compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | client-id | | | | | -| producer-metrics | connection-close-rate | The number of connections closed per second | client-id | | | | | -| producer-metrics | connection-close-total | The total number of connections closed | client-id | | | | | -| producer-metrics | connection-count | The current number of active connections. | client-id | | | | | -| producer-metrics | connection-creation-rate | The number of new connections established per second | client-id | | | | | -| producer-metrics | connection-creation-total | The total number of new connections established | client-id | | | | | -| producer-metrics | failed-authentication-rate | The number of connections with failed authentication per second | client-id | | | | | -| producer-metrics | failed-authentication-total | The total number of connections with failed authentication | client-id | | | | | -| producer-metrics | failed-reauthentication-rate | The number of failed re-authentication of connections per second | client-id | | | | | -| producer-metrics | failed-reauthentication-total | The total number of failed re-authentication of connections | client-id | | | | | -| producer-metrics | incoming-byte-rate | The number of bytes read off all sockets per second | client-id | | | | | -| producer-metrics | incoming-byte-total | The total number of bytes read off all sockets | client-id | | | | | -| producer-metrics | io-ratio | The fraction of time the I/O thread spent doing I/O | client-id | | | | | -| producer-metrics | io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | client-id | | | | | -| producer-metrics | io-wait-ratio | The fraction of time the I/O thread spent waiting | client-id | | | | | -| producer-metrics | io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | client-id | | | | | -| producer-metrics | io-waittime-total | The total time the I/O thread spent waiting | client-id | | | | | -| producer-metrics | iotime-total | The total time the I/O thread spent doing I/O | client-id | | | | | -| producer-metrics | metadata-age | The age in seconds of the current producer metadata being used. | client-id | | | | | -| producer-metrics | network-io-rate | The number of network operations (reads or writes) on all connections per second | client-id | | | | | -| producer-metrics | network-io-total | The total number of network operations (reads or writes) on all connections | client-id | | | | | -| producer-metrics | outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | client-id | messaging.kafka.producer.outgoing-bytes.rate | The average number of outgoing bytes sent per second to all servers. | by/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-metrics | outgoing-byte-total | The total number of outgoing bytes sent to all servers | client-id | | | | | -| producer-metrics | produce-throttle-time-avg | The average time in ms a request was throttled by a broker | client-id | | | | | -| producer-metrics | produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | client-id | | | | | -| producer-metrics | reauthentication-latency-avg | The average latency observed due to re-authentication | client-id | | | | | -| producer-metrics | reauthentication-latency-max | The max latency observed due to re-authentication | client-id | | | | | -| producer-metrics | record-error-rate | The average per-second number of record sends that resulted in errors | client-id | | | | | -| producer-metrics | record-error-total | The total number of record sends that resulted in errors | client-id | | | | | -| producer-metrics | record-queue-time-avg | The average time in ms record batches spent in the send buffer. | client-id | | | | | -| producer-metrics | record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | client-id | | | | | -| producer-metrics | record-retry-rate | The average per-second number of retried record sends | client-id | | | | | -| producer-metrics | record-retry-total | The total number of retried record sends | client-id | | | | | -| producer-metrics | record-send-rate | The average number of records sent per second. | client-id | | | | | -| producer-metrics | record-send-total | The total number of records sent. | client-id | | | | | -| producer-metrics | record-size-avg | The average record size | client-id | | | | | -| producer-metrics | record-size-max | The maximum record size | client-id | | | | | -| producer-metrics | records-per-request-avg | The average number of records per request. | client-id | | | | | -| producer-metrics | request-latency-avg | The average request latency in ms | client-id | | | | | -| producer-metrics | request-latency-max | The maximum request latency in ms | client-id | | | | | -| producer-metrics | request-rate | The number of requests sent per second | client-id | | | | | -| producer-metrics | request-size-avg | The average size of requests sent. | client-id | | | | | -| producer-metrics | request-size-max | The maximum size of any request sent. | client-id | | | | | -| producer-metrics | request-total | The total number of requests sent | client-id | | | | | -| producer-metrics | requests-in-flight | The current number of in-flight requests awaiting a response. | client-id | | | | | -| producer-metrics | response-rate | The number of responses received per second | client-id | messaging.kafka.producer.responses.rate | The average number of responses received per second. | {responses}/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-metrics | response-total | The total number of responses received | client-id | | | | | -| producer-metrics | select-rate | The number of times the I/O layer checked for new I/O to perform per second | client-id | | | | | -| producer-metrics | select-total | The total number of times the I/O layer checked for new I/O to perform | client-id | | | | | -| producer-metrics | successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | client-id | | | | | -| producer-metrics | successful-authentication-rate | The number of connections with successful authentication per second | client-id | | | | | -| producer-metrics | successful-authentication-total | The total number of connections with successful authentication | client-id | | | | | -| producer-metrics | successful-reauthentication-rate | The number of successful re-authentication of connections per second | client-id | | | | | -| producer-metrics | successful-reauthentication-total | The total number of successful re-authentication of connections | client-id | | | | | -| producer-metrics | waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | client-id | | | | | -| producer-node-metrics | incoming-byte-rate | The number of incoming bytes per second | client-id,node-id | | | | | -| producer-node-metrics | incoming-byte-total | The total number of incoming bytes | client-id,node-id | | | | | -| producer-node-metrics | outgoing-byte-rate | The number of outgoing bytes per second | client-id,node-id | | | | | -| producer-node-metrics | outgoing-byte-total | The total number of outgoing bytes | client-id,node-id | | | | | -| producer-node-metrics | request-latency-avg | | client-id,node-id | | | | | -| producer-node-metrics | request-latency-max | | client-id,node-id | | | | | -| producer-node-metrics | request-rate | The number of requests sent per second | client-id,node-id | | | | | -| producer-node-metrics | request-size-avg | The average size of requests sent. | client-id,node-id | | | | | -| producer-node-metrics | request-size-max | The maximum size of any request sent. | client-id,node-id | | | | | -| producer-node-metrics | request-total | The total number of requests sent | client-id,node-id | | | | | -| producer-node-metrics | response-rate | The number of responses received per second | client-id,node-id | | | | | -| producer-node-metrics | response-total | The total number of responses received | client-id,node-id | | | | | -| producer-topic-metrics | byte-rate | The average number of bytes sent per second for a topic. | client-id,topic | messaging.kafka.producer.bytes.rate | The average number of bytes sent per second for a specific topic. | by/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-topic-metrics | byte-total | The total number of bytes sent for a topic. | client-id,topic | | | | | -| producer-topic-metrics | compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | client-id,topic | messaging.kafka.producer.compression-ratio | The average compression ratio of record batches for a specific topic. | {compression} | DOUBLE_OBSERVABLE_GAUGE | -| producer-topic-metrics | record-error-rate | The average per-second number of record sends that resulted in errors for a topic | client-id,topic | messaging.kafka.producer.record-error.rate | The average per-second number of record sends that resulted in errors for a specific topic. | {errors}/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-topic-metrics | record-error-total | The total number of record sends that resulted in errors for a topic | client-id,topic | | | | | -| producer-topic-metrics | record-retry-rate | The average per-second number of retried record sends for a topic | client-id,topic | messaging.kafka.producer.record-retry.rate | The average per-second number of retried record sends for a specific topic. | {retries}/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-topic-metrics | record-retry-total | The total number of retried record sends for a topic | client-id,topic | | | | | -| producer-topic-metrics | record-send-rate | The average number of records sent per second for a topic. | client-id,topic | messaging.kafka.producer.record-sent.rate | The average number of records sent per second for a specific topic. | {records_sent}/s | DOUBLE_OBSERVABLE_GAUGE | -| producer-topic-metrics | record-send-total | The total number of records sent for a topic. | client-id,topic | | | | | \ No newline at end of file +| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | +|---|--------------|-------------|----------------|-----------------|------------------------|-----------------| +| 1 | app-info | commit-id | client-id | | | | +| 2 | app-info | start-time-ms | client-id | | | | +| 3 | app-info | version | client-id | | | | +| 4 | consumer-coordinator-metrics | assigned-partitions | client-id | consumer-coordinator-metrics.assigned-partitions | The number of partitions currently assigned to this consumer | DOUBLE_OBSERVABLE_GAUGE | +| 5 | consumer-coordinator-metrics | commit-latency-avg | client-id | consumer-coordinator-metrics.commit-latency-avg | The average time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | +| 6 | consumer-coordinator-metrics | commit-latency-max | client-id | consumer-coordinator-metrics.commit-latency-max | The max time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | +| 7 | consumer-coordinator-metrics | commit-rate | client-id | consumer-coordinator-metrics.commit-rate | The number of commit calls per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 8 | consumer-coordinator-metrics | commit-total | client-id | consumer-coordinator-metrics.commit-total | The total number of commit calls | DOUBLE_OBSERVABLE_COUNTER | +| 9 | consumer-coordinator-metrics | failed-rebalance-rate-per-hour | client-id | consumer-coordinator-metrics.failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 10 | consumer-coordinator-metrics | failed-rebalance-total | client-id | consumer-coordinator-metrics.failed-rebalance-total | The total number of failed rebalance events | DOUBLE_OBSERVABLE_COUNTER | +| 11 | consumer-coordinator-metrics | heartbeat-rate | client-id | consumer-coordinator-metrics.heartbeat-rate | The number of heartbeats per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 12 | consumer-coordinator-metrics | heartbeat-response-time-max | client-id | consumer-coordinator-metrics.heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | DOUBLE_OBSERVABLE_GAUGE | +| 13 | consumer-coordinator-metrics | heartbeat-total | client-id | consumer-coordinator-metrics.heartbeat-total | The total number of heartbeats | DOUBLE_OBSERVABLE_COUNTER | +| 14 | consumer-coordinator-metrics | join-rate | client-id | consumer-coordinator-metrics.join-rate | The number of group joins per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 15 | consumer-coordinator-metrics | join-time-avg | client-id | consumer-coordinator-metrics.join-time-avg | The average time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | +| 16 | consumer-coordinator-metrics | join-time-max | client-id | consumer-coordinator-metrics.join-time-max | The max time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | +| 17 | consumer-coordinator-metrics | join-total | client-id | consumer-coordinator-metrics.join-total | The total number of group joins | DOUBLE_OBSERVABLE_COUNTER | +| 18 | consumer-coordinator-metrics | last-heartbeat-seconds-ago | client-id | consumer-coordinator-metrics.last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | DOUBLE_OBSERVABLE_GAUGE | +| 19 | consumer-coordinator-metrics | last-rebalance-seconds-ago | client-id | consumer-coordinator-metrics.last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | DOUBLE_OBSERVABLE_GAUGE | +| 20 | consumer-coordinator-metrics | partition-assigned-latency-avg | client-id | consumer-coordinator-metrics.partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 21 | consumer-coordinator-metrics | partition-assigned-latency-max | client-id | consumer-coordinator-metrics.partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 22 | consumer-coordinator-metrics | partition-lost-latency-avg | client-id | consumer-coordinator-metrics.partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 23 | consumer-coordinator-metrics | partition-lost-latency-max | client-id | consumer-coordinator-metrics.partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 24 | consumer-coordinator-metrics | partition-revoked-latency-avg | client-id | consumer-coordinator-metrics.partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 25 | consumer-coordinator-metrics | partition-revoked-latency-max | client-id | consumer-coordinator-metrics.partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 26 | consumer-coordinator-metrics | rebalance-latency-avg | client-id | consumer-coordinator-metrics.rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | +| 27 | consumer-coordinator-metrics | rebalance-latency-max | client-id | consumer-coordinator-metrics.rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | +| 28 | consumer-coordinator-metrics | rebalance-latency-total | client-id | consumer-coordinator-metrics.rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | DOUBLE_OBSERVABLE_COUNTER | +| 29 | consumer-coordinator-metrics | rebalance-rate-per-hour | client-id | consumer-coordinator-metrics.rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 30 | consumer-coordinator-metrics | rebalance-total | client-id | consumer-coordinator-metrics.rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_COUNTER | +| 31 | consumer-coordinator-metrics | sync-rate | client-id | consumer-coordinator-metrics.sync-rate | The number of group syncs per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 32 | consumer-coordinator-metrics | sync-time-avg | client-id | consumer-coordinator-metrics.sync-time-avg | The average time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | +| 33 | consumer-coordinator-metrics | sync-time-max | client-id | consumer-coordinator-metrics.sync-time-max | The max time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | +| 34 | consumer-coordinator-metrics | sync-total | client-id | consumer-coordinator-metrics.sync-total | The total number of group syncs | DOUBLE_OBSERVABLE_COUNTER | +| 35 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id | consumer-fetch-manager-metrics.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 36 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id,topic | consumer-fetch-manager-metrics.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 37 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id | consumer-fetch-manager-metrics.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | +| 38 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id,topic | consumer-fetch-manager-metrics.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | +| 39 | consumer-fetch-manager-metrics | fetch-latency-avg | client-id | consumer-fetch-manager-metrics.fetch-latency-avg | The average time taken for a fetch request. | DOUBLE_OBSERVABLE_GAUGE | +| 40 | consumer-fetch-manager-metrics | fetch-latency-max | client-id | consumer-fetch-manager-metrics.fetch-latency-max | The max time taken for any fetch request. | DOUBLE_OBSERVABLE_GAUGE | +| 41 | consumer-fetch-manager-metrics | fetch-rate | client-id | consumer-fetch-manager-metrics.fetch-rate | The number of fetch requests per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 42 | consumer-fetch-manager-metrics | fetch-size-avg | client-id | consumer-fetch-manager-metrics.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 43 | consumer-fetch-manager-metrics | fetch-size-avg | client-id,topic | consumer-fetch-manager-metrics.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 44 | consumer-fetch-manager-metrics | fetch-size-max | client-id | consumer-fetch-manager-metrics.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 45 | consumer-fetch-manager-metrics | fetch-size-max | client-id,topic | consumer-fetch-manager-metrics.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 46 | consumer-fetch-manager-metrics | fetch-throttle-time-avg | client-id | consumer-fetch-manager-metrics.fetch-throttle-time-avg | The average throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | +| 47 | consumer-fetch-manager-metrics | fetch-throttle-time-max | client-id | consumer-fetch-manager-metrics.fetch-throttle-time-max | The maximum throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | +| 48 | consumer-fetch-manager-metrics | fetch-total | client-id | consumer-fetch-manager-metrics.fetch-total | The total number of fetch requests. | DOUBLE_OBSERVABLE_COUNTER | +| 49 | consumer-fetch-manager-metrics | preferred-read-replica | client-id,topic,partition | | | | +| 50 | consumer-fetch-manager-metrics | records-consumed-rate | client-id | consumer-fetch-manager-metrics.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 51 | consumer-fetch-manager-metrics | records-consumed-rate | client-id,topic | consumer-fetch-manager-metrics.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 52 | consumer-fetch-manager-metrics | records-consumed-total | client-id,topic | consumer-fetch-manager-metrics.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | +| 53 | consumer-fetch-manager-metrics | records-consumed-total | client-id | consumer-fetch-manager-metrics.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | +| 54 | consumer-fetch-manager-metrics | records-lag | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag | The latest lag of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 55 | consumer-fetch-manager-metrics | records-lag-avg | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag-avg | The average lag of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 56 | consumer-fetch-manager-metrics | records-lag-max | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 57 | consumer-fetch-manager-metrics | records-lag-max | client-id | consumer-fetch-manager-metrics.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 58 | consumer-fetch-manager-metrics | records-lead | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead | The latest lead of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 59 | consumer-fetch-manager-metrics | records-lead-avg | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead-avg | The average lead of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 60 | consumer-fetch-manager-metrics | records-lead-min | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 61 | consumer-fetch-manager-metrics | records-lead-min | client-id | consumer-fetch-manager-metrics.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 62 | consumer-fetch-manager-metrics | records-per-request-avg | client-id,topic | consumer-fetch-manager-metrics.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | +| 63 | consumer-fetch-manager-metrics | records-per-request-avg | client-id | consumer-fetch-manager-metrics.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | +| 64 | consumer-metrics | connection-close-rate | client-id | consumer-metrics.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 65 | consumer-metrics | connection-close-total | client-id | consumer-metrics.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | +| 66 | consumer-metrics | connection-count | client-id | consumer-metrics.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | +| 67 | consumer-metrics | connection-creation-rate | client-id | consumer-metrics.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 68 | consumer-metrics | connection-creation-total | client-id | consumer-metrics.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | +| 69 | consumer-metrics | failed-authentication-rate | client-id | consumer-metrics.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 70 | consumer-metrics | failed-authentication-total | client-id | consumer-metrics.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | +| 71 | consumer-metrics | failed-reauthentication-rate | client-id | consumer-metrics.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 72 | consumer-metrics | failed-reauthentication-total | client-id | consumer-metrics.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 73 | consumer-metrics | incoming-byte-rate | client-id | consumer-metrics.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 74 | consumer-metrics | incoming-byte-total | client-id | consumer-metrics.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 75 | consumer-metrics | io-ratio | client-id | consumer-metrics.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 76 | consumer-metrics | io-time-ns-avg | client-id | consumer-metrics.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 77 | consumer-metrics | io-wait-ratio | client-id | consumer-metrics.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 78 | consumer-metrics | io-wait-time-ns-avg | client-id | consumer-metrics.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 79 | consumer-metrics | io-waittime-total | client-id | consumer-metrics.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | +| 80 | consumer-metrics | iotime-total | client-id | consumer-metrics.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | +| 81 | consumer-metrics | last-poll-seconds-ago | client-id | consumer-metrics.last-poll-seconds-ago | The number of seconds since the last poll() invocation. | DOUBLE_OBSERVABLE_GAUGE | +| 82 | consumer-metrics | network-io-rate | client-id | consumer-metrics.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 83 | consumer-metrics | network-io-total | client-id | consumer-metrics.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | +| 84 | consumer-metrics | outgoing-byte-rate | client-id | consumer-metrics.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 85 | consumer-metrics | outgoing-byte-total | client-id | consumer-metrics.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 86 | consumer-metrics | poll-idle-ratio-avg | client-id | consumer-metrics.poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | DOUBLE_OBSERVABLE_GAUGE | +| 87 | consumer-metrics | reauthentication-latency-avg | client-id | consumer-metrics.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 88 | consumer-metrics | reauthentication-latency-max | client-id | consumer-metrics.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 89 | consumer-metrics | request-rate | client-id | consumer-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 90 | consumer-metrics | request-size-avg | client-id | consumer-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 91 | consumer-metrics | request-size-max | client-id | consumer-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 92 | consumer-metrics | request-total | client-id | consumer-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 93 | consumer-metrics | response-rate | client-id | consumer-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 94 | consumer-metrics | response-total | client-id | consumer-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 95 | consumer-metrics | select-rate | client-id | consumer-metrics.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 96 | consumer-metrics | select-total | client-id | consumer-metrics.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | +| 97 | consumer-metrics | successful-authentication-no-reauth-total | client-id | consumer-metrics.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | +| 98 | consumer-metrics | successful-authentication-rate | client-id | consumer-metrics.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 99 | consumer-metrics | successful-authentication-total | client-id | consumer-metrics.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | +| 100 | consumer-metrics | successful-reauthentication-rate | client-id | consumer-metrics.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 101 | consumer-metrics | successful-reauthentication-total | client-id | consumer-metrics.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 102 | consumer-metrics | time-between-poll-avg | client-id | consumer-metrics.time-between-poll-avg | The average delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | +| 103 | consumer-metrics | time-between-poll-max | client-id | consumer-metrics.time-between-poll-max | The max delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | +| 104 | consumer-node-metrics | incoming-byte-rate | client-id,node-id | consumer-node-metrics.incoming-byte-rate | The number of incoming bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 105 | consumer-node-metrics | incoming-byte-total | client-id,node-id | consumer-node-metrics.incoming-byte-total | The total number of incoming bytes | DOUBLE_OBSERVABLE_COUNTER | +| 106 | consumer-node-metrics | outgoing-byte-rate | client-id,node-id | consumer-node-metrics.outgoing-byte-rate | The number of outgoing bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 107 | consumer-node-metrics | outgoing-byte-total | client-id,node-id | consumer-node-metrics.outgoing-byte-total | The total number of outgoing bytes | DOUBLE_OBSERVABLE_COUNTER | +| 108 | consumer-node-metrics | request-latency-avg | client-id,node-id | consumer-node-metrics.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | +| 109 | consumer-node-metrics | request-latency-max | client-id,node-id | consumer-node-metrics.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | +| 110 | consumer-node-metrics | request-rate | client-id,node-id | consumer-node-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 111 | consumer-node-metrics | request-size-avg | client-id,node-id | consumer-node-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 112 | consumer-node-metrics | request-size-max | client-id,node-id | consumer-node-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 113 | consumer-node-metrics | request-total | client-id,node-id | consumer-node-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 114 | consumer-node-metrics | response-rate | client-id,node-id | consumer-node-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 115 | consumer-node-metrics | response-total | client-id,node-id | consumer-node-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 116 | kafka-metrics-count | count | client-id | kafka-metrics-count.count | total number of registered metrics | DOUBLE_OBSERVABLE_GAUGE | +| 117 | producer-metrics | batch-size-avg | client-id | producer-metrics.batch-size-avg | The average number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | +| 118 | producer-metrics | batch-size-max | client-id | producer-metrics.batch-size-max | The max number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | +| 119 | producer-metrics | batch-split-rate | client-id | producer-metrics.batch-split-rate | The average number of batch splits per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 120 | producer-metrics | batch-split-total | client-id | producer-metrics.batch-split-total | The total number of batch splits | DOUBLE_OBSERVABLE_COUNTER | +| 121 | producer-metrics | buffer-available-bytes | client-id | producer-metrics.buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | DOUBLE_OBSERVABLE_GAUGE | +| 122 | producer-metrics | buffer-exhausted-rate | client-id | producer-metrics.buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 123 | producer-metrics | buffer-exhausted-total | client-id | producer-metrics.buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_COUNTER | +| 124 | producer-metrics | buffer-total-bytes | client-id | producer-metrics.buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | DOUBLE_OBSERVABLE_GAUGE | +| 125 | producer-metrics | bufferpool-wait-ratio | client-id | producer-metrics.bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 126 | producer-metrics | bufferpool-wait-time-total | client-id | producer-metrics.bufferpool-wait-time-total | The total time an appender waits for space allocation. | DOUBLE_OBSERVABLE_COUNTER | +| 127 | producer-metrics | compression-rate-avg | client-id | producer-metrics.compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | +| 128 | producer-metrics | connection-close-rate | client-id | producer-metrics.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 129 | producer-metrics | connection-close-total | client-id | producer-metrics.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | +| 130 | producer-metrics | connection-count | client-id | producer-metrics.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | +| 131 | producer-metrics | connection-creation-rate | client-id | producer-metrics.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 132 | producer-metrics | connection-creation-total | client-id | producer-metrics.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | +| 133 | producer-metrics | failed-authentication-rate | client-id | producer-metrics.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 134 | producer-metrics | failed-authentication-total | client-id | producer-metrics.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | +| 135 | producer-metrics | failed-reauthentication-rate | client-id | producer-metrics.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 136 | producer-metrics | failed-reauthentication-total | client-id | producer-metrics.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 137 | producer-metrics | incoming-byte-rate | client-id | producer-metrics.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 138 | producer-metrics | incoming-byte-total | client-id | producer-metrics.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 139 | producer-metrics | io-ratio | client-id | producer-metrics.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 140 | producer-metrics | io-time-ns-avg | client-id | producer-metrics.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 141 | producer-metrics | io-wait-ratio | client-id | producer-metrics.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 142 | producer-metrics | io-wait-time-ns-avg | client-id | producer-metrics.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 143 | producer-metrics | io-waittime-total | client-id | producer-metrics.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | +| 144 | producer-metrics | iotime-total | client-id | producer-metrics.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | +| 145 | producer-metrics | metadata-age | client-id | producer-metrics.metadata-age | The age in seconds of the current producer metadata being used. | DOUBLE_OBSERVABLE_GAUGE | +| 146 | producer-metrics | network-io-rate | client-id | producer-metrics.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 147 | producer-metrics | network-io-total | client-id | producer-metrics.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | +| 148 | producer-metrics | outgoing-byte-rate | client-id | producer-metrics.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 149 | producer-metrics | outgoing-byte-total | client-id | producer-metrics.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 150 | producer-metrics | produce-throttle-time-avg | client-id | producer-metrics.produce-throttle-time-avg | The average time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | +| 151 | producer-metrics | produce-throttle-time-max | client-id | producer-metrics.produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | +| 152 | producer-metrics | reauthentication-latency-avg | client-id | producer-metrics.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 153 | producer-metrics | reauthentication-latency-max | client-id | producer-metrics.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 154 | producer-metrics | record-error-rate | client-id | producer-metrics.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 155 | producer-metrics | record-error-total | client-id | producer-metrics.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | +| 156 | producer-metrics | record-queue-time-avg | client-id | producer-metrics.record-queue-time-avg | The average time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | +| 157 | producer-metrics | record-queue-time-max | client-id | producer-metrics.record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | +| 158 | producer-metrics | record-retry-rate | client-id | producer-metrics.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 159 | producer-metrics | record-retry-total | client-id | producer-metrics.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | +| 160 | producer-metrics | record-send-rate | client-id | producer-metrics.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 161 | producer-metrics | record-send-total | client-id | producer-metrics.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | +| 162 | producer-metrics | record-size-avg | client-id | producer-metrics.record-size-avg | The average record size | DOUBLE_OBSERVABLE_GAUGE | +| 163 | producer-metrics | record-size-max | client-id | producer-metrics.record-size-max | The maximum record size | DOUBLE_OBSERVABLE_GAUGE | +| 164 | producer-metrics | records-per-request-avg | client-id | producer-metrics.records-per-request-avg | The average number of records per request. | DOUBLE_OBSERVABLE_GAUGE | +| 165 | producer-metrics | request-latency-avg | client-id | producer-metrics.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 166 | producer-metrics | request-latency-max | client-id | producer-metrics.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 167 | producer-metrics | request-rate | client-id | producer-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 168 | producer-metrics | request-size-avg | client-id | producer-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 169 | producer-metrics | request-size-max | client-id | producer-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 170 | producer-metrics | request-total | client-id | producer-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 171 | producer-metrics | requests-in-flight | client-id | producer-metrics.requests-in-flight | The current number of in-flight requests awaiting a response. | DOUBLE_OBSERVABLE_GAUGE | +| 172 | producer-metrics | response-rate | client-id | producer-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 173 | producer-metrics | response-total | client-id | producer-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 174 | producer-metrics | select-rate | client-id | producer-metrics.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 175 | producer-metrics | select-total | client-id | producer-metrics.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | +| 176 | producer-metrics | successful-authentication-no-reauth-total | client-id | producer-metrics.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | +| 177 | producer-metrics | successful-authentication-rate | client-id | producer-metrics.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 178 | producer-metrics | successful-authentication-total | client-id | producer-metrics.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | +| 179 | producer-metrics | successful-reauthentication-rate | client-id | producer-metrics.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 180 | producer-metrics | successful-reauthentication-total | client-id | producer-metrics.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 181 | producer-metrics | waiting-threads | client-id | producer-metrics.waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | DOUBLE_OBSERVABLE_GAUGE | +| 182 | producer-node-metrics | incoming-byte-rate | client-id,node-id | producer-node-metrics.incoming-byte-rate | The number of incoming bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 183 | producer-node-metrics | incoming-byte-total | client-id,node-id | producer-node-metrics.incoming-byte-total | The total number of incoming bytes | DOUBLE_OBSERVABLE_COUNTER | +| 184 | producer-node-metrics | outgoing-byte-rate | client-id,node-id | producer-node-metrics.outgoing-byte-rate | The number of outgoing bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 185 | producer-node-metrics | outgoing-byte-total | client-id,node-id | producer-node-metrics.outgoing-byte-total | The total number of outgoing bytes | DOUBLE_OBSERVABLE_COUNTER | +| 186 | producer-node-metrics | request-latency-avg | client-id,node-id | producer-node-metrics.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | +| 187 | producer-node-metrics | request-latency-max | client-id,node-id | producer-node-metrics.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | +| 188 | producer-node-metrics | request-rate | client-id,node-id | producer-node-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 189 | producer-node-metrics | request-size-avg | client-id,node-id | producer-node-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 190 | producer-node-metrics | request-size-max | client-id,node-id | producer-node-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 191 | producer-node-metrics | request-total | client-id,node-id | producer-node-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 192 | producer-node-metrics | response-rate | client-id,node-id | producer-node-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 193 | producer-node-metrics | response-total | client-id,node-id | producer-node-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 194 | producer-topic-metrics | byte-rate | client-id,topic | producer-topic-metrics.byte-rate | The average number of bytes sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 195 | producer-topic-metrics | byte-total | client-id,topic | producer-topic-metrics.byte-total | The total number of bytes sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | +| 196 | producer-topic-metrics | compression-rate | client-id,topic | producer-topic-metrics.compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | +| 197 | producer-topic-metrics | record-error-rate | client-id,topic | producer-topic-metrics.record-error-rate | The average per-second number of record sends that resulted in errors for a topic | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 198 | producer-topic-metrics | record-error-total | client-id,topic | producer-topic-metrics.record-error-total | The total number of record sends that resulted in errors for a topic | DOUBLE_OBSERVABLE_COUNTER | +| 199 | producer-topic-metrics | record-retry-rate | client-id,topic | producer-topic-metrics.record-retry-rate | The average per-second number of retried record sends for a topic | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 200 | producer-topic-metrics | record-retry-total | client-id,topic | producer-topic-metrics.record-retry-total | The total number of retried record sends for a topic | DOUBLE_OBSERVABLE_COUNTER | +| 201 | producer-topic-metrics | record-send-rate | client-id,topic | producer-topic-metrics.record-send-rate | The average number of records sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 202 | producer-topic-metrics | record-send-total | client-id,topic | producer-topic-metrics.record-send-total | The total number of records sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index dbd87631becd..8f5c59212ec9 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -5,16 +5,22 @@ package io.opentelemetry.instrumentation.kafkaclients; +import static java.util.stream.Collectors.toSet; +import static org.assertj.core.api.Assertions.assertThat; + import com.salesforce.kafka.test.junit5.SharedKafkaTestResource; import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension; import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension; +import io.opentelemetry.sdk.metrics.data.MetricData; import java.nio.charset.StandardCharsets; import java.time.Instant; import java.util.Arrays; import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.producer.KafkaProducer; @@ -42,32 +48,202 @@ void observeMetrics() { produceRecords(); consumeRecords(); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", - "messaging.kafka.producer.outgoing-bytes.rate", - unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", "messaging.kafka.producer.responses.rate", unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", "messaging.kafka.producer.bytes.rate", unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", - "messaging.kafka.producer.compression-ratio", - unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", - "messaging.kafka.producer.record-error.rate", - unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", - "messaging.kafka.producer.record-retry.rate", - unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", - "messaging.kafka.producer.record-sent.rate", - unused -> {}); - testing.waitAndAssertMetrics( - "io.opentelemetry.kafka-clients", "messaging.kafka.consumer.lag", unused -> {}); + Set expectedMetricNames = + new HashSet<>( + Arrays.asList( + "consumer-coordinator-metrics.assigned-partitions", + "consumer-coordinator-metrics.commit-latency-avg", + "consumer-coordinator-metrics.commit-latency-max", + "consumer-coordinator-metrics.commit-rate", + "consumer-coordinator-metrics.commit-total", + "consumer-coordinator-metrics.failed-rebalance-rate-per-hour", + "consumer-coordinator-metrics.failed-rebalance-total", + "consumer-coordinator-metrics.heartbeat-rate", + "consumer-coordinator-metrics.heartbeat-response-time-max", + "consumer-coordinator-metrics.heartbeat-total", + "consumer-coordinator-metrics.join-rate", + "consumer-coordinator-metrics.join-time-avg", + "consumer-coordinator-metrics.join-time-max", + "consumer-coordinator-metrics.join-total", + "consumer-coordinator-metrics.last-heartbeat-seconds-ago", + "consumer-coordinator-metrics.last-rebalance-seconds-ago", + "consumer-coordinator-metrics.partition-assigned-latency-avg", + "consumer-coordinator-metrics.partition-assigned-latency-max", + "consumer-coordinator-metrics.partition-lost-latency-avg", + "consumer-coordinator-metrics.partition-lost-latency-max", + "consumer-coordinator-metrics.partition-revoked-latency-avg", + "consumer-coordinator-metrics.partition-revoked-latency-max", + "consumer-coordinator-metrics.rebalance-latency-avg", + "consumer-coordinator-metrics.rebalance-latency-max", + "consumer-coordinator-metrics.rebalance-latency-total", + "consumer-coordinator-metrics.rebalance-rate-per-hour", + "consumer-coordinator-metrics.rebalance-total", + "consumer-coordinator-metrics.sync-rate", + "consumer-coordinator-metrics.sync-time-avg", + "consumer-coordinator-metrics.sync-time-max", + "consumer-coordinator-metrics.sync-total", + "consumer-fetch-manager-metrics.bytes-consumed-rate", + "consumer-fetch-manager-metrics.bytes-consumed-total", + "consumer-fetch-manager-metrics.fetch-latency-avg", + "consumer-fetch-manager-metrics.fetch-latency-max", + "consumer-fetch-manager-metrics.fetch-rate", + "consumer-fetch-manager-metrics.fetch-size-avg", + "consumer-fetch-manager-metrics.fetch-size-max", + "consumer-fetch-manager-metrics.fetch-throttle-time-avg", + "consumer-fetch-manager-metrics.fetch-throttle-time-max", + "consumer-fetch-manager-metrics.fetch-total", + "consumer-fetch-manager-metrics.records-consumed-rate", + "consumer-fetch-manager-metrics.records-consumed-total", + "consumer-fetch-manager-metrics.records-lag", + "consumer-fetch-manager-metrics.records-lag-avg", + "consumer-fetch-manager-metrics.records-lag-max", + "consumer-fetch-manager-metrics.records-lead", + "consumer-fetch-manager-metrics.records-lead-avg", + "consumer-fetch-manager-metrics.records-lead-min", + "consumer-fetch-manager-metrics.records-per-request-avg", + "consumer-metrics.connection-close-rate", + "consumer-metrics.connection-close-total", + "consumer-metrics.connection-count", + "consumer-metrics.connection-creation-rate", + "consumer-metrics.connection-creation-total", + "consumer-metrics.failed-authentication-rate", + "consumer-metrics.failed-authentication-total", + "consumer-metrics.failed-reauthentication-rate", + "consumer-metrics.failed-reauthentication-total", + "consumer-metrics.incoming-byte-rate", + "consumer-metrics.incoming-byte-total", + "consumer-metrics.io-ratio", + "consumer-metrics.io-time-ns-avg", + "consumer-metrics.io-wait-ratio", + "consumer-metrics.io-wait-time-ns-avg", + "consumer-metrics.io-waittime-total", + "consumer-metrics.iotime-total", + "consumer-metrics.last-poll-seconds-ago", + "consumer-metrics.network-io-rate", + "consumer-metrics.network-io-total", + "consumer-metrics.outgoing-byte-rate", + "consumer-metrics.outgoing-byte-total", + "consumer-metrics.poll-idle-ratio-avg", + "consumer-metrics.reauthentication-latency-avg", + "consumer-metrics.reauthentication-latency-max", + "consumer-metrics.request-rate", + "consumer-metrics.request-size-avg", + "consumer-metrics.request-size-max", + "consumer-metrics.request-total", + "consumer-metrics.response-rate", + "consumer-metrics.response-total", + "consumer-metrics.select-rate", + "consumer-metrics.select-total", + "consumer-metrics.successful-authentication-no-reauth-total", + "consumer-metrics.successful-authentication-rate", + "consumer-metrics.successful-authentication-total", + "consumer-metrics.successful-reauthentication-rate", + "consumer-metrics.successful-reauthentication-total", + "consumer-metrics.time-between-poll-avg", + "consumer-metrics.time-between-poll-max", + "consumer-node-metrics.incoming-byte-rate", + "consumer-node-metrics.incoming-byte-total", + "consumer-node-metrics.outgoing-byte-rate", + "consumer-node-metrics.outgoing-byte-total", + "consumer-node-metrics.request-latency-avg", + "consumer-node-metrics.request-latency-max", + "consumer-node-metrics.request-rate", + "consumer-node-metrics.request-size-avg", + "consumer-node-metrics.request-size-max", + "consumer-node-metrics.request-total", + "consumer-node-metrics.response-rate", + "consumer-node-metrics.response-total", + "kafka-metrics-count.count", + "producer-metrics.batch-size-avg", + "producer-metrics.batch-size-max", + "producer-metrics.batch-split-rate", + "producer-metrics.batch-split-total", + "producer-metrics.buffer-available-bytes", + "producer-metrics.buffer-exhausted-rate", + "producer-metrics.buffer-exhausted-total", + "producer-metrics.buffer-total-bytes", + "producer-metrics.bufferpool-wait-ratio", + "producer-metrics.bufferpool-wait-time-total", + "producer-metrics.compression-rate-avg", + "producer-metrics.connection-close-rate", + "producer-metrics.connection-close-total", + "producer-metrics.connection-count", + "producer-metrics.connection-creation-rate", + "producer-metrics.connection-creation-total", + "producer-metrics.failed-authentication-rate", + "producer-metrics.failed-authentication-total", + "producer-metrics.failed-reauthentication-rate", + "producer-metrics.failed-reauthentication-total", + "producer-metrics.incoming-byte-rate", + "producer-metrics.incoming-byte-total", + "producer-metrics.io-ratio", + "producer-metrics.io-time-ns-avg", + "producer-metrics.io-wait-ratio", + "producer-metrics.io-wait-time-ns-avg", + "producer-metrics.io-waittime-total", + "producer-metrics.iotime-total", + "producer-metrics.metadata-age", + "producer-metrics.network-io-rate", + "producer-metrics.network-io-total", + "producer-metrics.outgoing-byte-rate", + "producer-metrics.outgoing-byte-total", + "producer-metrics.produce-throttle-time-avg", + "producer-metrics.produce-throttle-time-max", + "producer-metrics.reauthentication-latency-avg", + "producer-metrics.reauthentication-latency-max", + "producer-metrics.record-error-rate", + "producer-metrics.record-error-total", + "producer-metrics.record-queue-time-avg", + "producer-metrics.record-queue-time-max", + "producer-metrics.record-retry-rate", + "producer-metrics.record-retry-total", + "producer-metrics.record-send-rate", + "producer-metrics.record-send-total", + "producer-metrics.record-size-avg", + "producer-metrics.record-size-max", + "producer-metrics.records-per-request-avg", + "producer-metrics.request-latency-avg", + "producer-metrics.request-latency-max", + "producer-metrics.request-rate", + "producer-metrics.request-size-avg", + "producer-metrics.request-size-max", + "producer-metrics.request-total", + "producer-metrics.requests-in-flight", + "producer-metrics.response-rate", + "producer-metrics.response-total", + "producer-metrics.select-rate", + "producer-metrics.select-total", + "producer-metrics.successful-authentication-no-reauth-total", + "producer-metrics.successful-authentication-rate", + "producer-metrics.successful-authentication-total", + "producer-metrics.successful-reauthentication-rate", + "producer-metrics.successful-reauthentication-total", + "producer-metrics.waiting-threads", + "producer-node-metrics.incoming-byte-rate", + "producer-node-metrics.incoming-byte-total", + "producer-node-metrics.outgoing-byte-rate", + "producer-node-metrics.outgoing-byte-total", + "producer-node-metrics.request-latency-avg", + "producer-node-metrics.request-latency-max", + "producer-node-metrics.request-rate", + "producer-node-metrics.request-size-avg", + "producer-node-metrics.request-size-max", + "producer-node-metrics.request-total", + "producer-node-metrics.response-rate", + "producer-node-metrics.response-total", + "producer-topic-metrics.byte-rate", + "producer-topic-metrics.byte-total", + "producer-topic-metrics.compression-rate", + "producer-topic-metrics.record-error-rate", + "producer-topic-metrics.record-error-total", + "producer-topic-metrics.record-retry-rate", + "producer-topic-metrics.record-retry-total", + "producer-topic-metrics.record-send-rate", + "producer-topic-metrics.record-send-total")); + + Set metricNames = testing.metrics().stream().map(MetricData::getName).collect(toSet()); + + assertThat(metricNames).containsAll(expectedMetricNames); // Print mapping table OpenTelemetryKafkaMetrics.printMappingTable(); diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java new file mode 100644 index 000000000000..d69b0b0770a7 --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java @@ -0,0 +1,39 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.instrumentation.kafkaclients; + +import com.google.auto.value.AutoValue; + +/** A description of an OpenTelemetry metric instrument. */ +@AutoValue +abstract class InstrumentDescriptor { + + static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE = "DOUBLE_OBSERVABLE_GAUGE"; + static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER = + "DOUBLE_OBSERVABLE_UP_DOWN_COUNTER"; + static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER = "DOUBLE_OBSERVABLE_COUNTER"; + + abstract String getName(); + + abstract String getDescription(); + + abstract String getInstrumentType(); + + static InstrumentDescriptor createDoubleGauge(String name, String description) { + return new AutoValue_InstrumentDescriptor( + name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + } + + static InstrumentDescriptor createDoubleCounter(String name, String description) { + return new AutoValue_InstrumentDescriptor( + name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER); + } + + static InstrumentDescriptor createDoubleUpDownCounter(String name, String description) { + return new AutoValue_InstrumentDescriptor( + name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER); + } +} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java index 598d821d9db4..71c5166abcd9 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java @@ -5,101 +5,39 @@ package io.opentelemetry.instrumentation.kafkaclients; +import com.google.auto.value.AutoValue; import java.util.Set; import javax.annotation.Nullable; -import org.apache.kafka.common.MetricName; import org.apache.kafka.common.metrics.KafkaMetric; +import org.apache.kafka.common.metrics.Measurable; -/** - * A value class collecting the identifying fields of a kafka {@link MetricName}. - * - *

Note: {@link #description} is included in {@link #toString()} but omitted from {@link - * #equals(Object)} and {@link #hashCode()}. - */ -class KafkaMetricId { - - private final String name; - private final String group; - private final String description; - private final Set tagKeys; - - private KafkaMetricId(String name, String group, String description, Set tagKeys) { - this.name = name; - this.group = group; - this.description = description; - this.tagKeys = tagKeys; - } - - static KafkaMetricId create(KafkaMetric kafkaMetric) { - return new KafkaMetricId( - kafkaMetric.metricName().name(), - kafkaMetric.metricName().group(), - kafkaMetric.metricName().description(), - kafkaMetric.metricName().tags().keySet()); - } - - static KafkaMetricId create(String name, String group, Set tagKeys) { - return new KafkaMetricId(name, group, "", tagKeys); - } +/** A value class collecting the identifying fields of a kafka {@link KafkaMetric}. */ +@AutoValue +abstract class KafkaMetricId { - String getName() { - return name; - } + abstract String getGroup(); - String getGroup() { - return group; - } + abstract String getName(); - String getDescription() { - return description; - } + abstract String getDescription(); - Set getTagKeys() { - return tagKeys; - } + @Nullable + abstract Class getMeasureable(); - @Override - public String toString() { - return "KafkaMetricId{" - + "name=" - + name - + ", " - + "group=" - + group - + ", " - + "description=" - + description - + ", " - + "tagKeys=" - + tagKeys - + "}"; - } + abstract Set getAttributeKeys(); - @Override - public boolean equals(@Nullable Object o) { - if (o == this) { - return true; - } - if (o instanceof KafkaMetricId) { - KafkaMetricId that = (KafkaMetricId) o; - // Omit description from equality - return this.name.equals(that.getName()) - && this.group.equals(that.getGroup()) - && this.tagKeys.equals(that.getTagKeys()); + static KafkaMetricId create(KafkaMetric kafkaMetric) { + Class measureable; + try { + measureable = kafkaMetric.measurable().getClass(); + } catch (IllegalStateException e) { + measureable = null; } - return false; - } - - @Override - public int hashCode() { - // Omit description from hashcode - int hash = 1; - hash *= 1000003; - hash ^= name.hashCode(); - hash *= 1000003; - hash ^= group.hashCode(); - hash *= 1000003; - hash ^= tagKeys.hashCode(); - return hash; + return new AutoValue_KafkaMetricId( + kafkaMetric.metricName().group(), + kafkaMetric.metricName().name(), + kafkaMetric.metricName().description(), + measureable, + kafkaMetric.metricName().tags().keySet()); } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index 385fe26a200e..4911921f297e 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -5,105 +5,74 @@ package io.opentelemetry.instrumentation.kafkaclients; -import java.util.Arrays; -import java.util.Collections; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER; + import java.util.HashMap; -import java.util.HashSet; import java.util.Map; -import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import javax.annotation.Nullable; /** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */ class KafkaMetricRegistry { - private static final String unitBytesPerSecond = "by/s"; - - private static final Map registry = new HashMap<>(); + private static final Map, String> measureableToInstrumentType = new HashMap<>(); + private static final Map descriptionCache = new ConcurrentHashMap<>(); static { - registerProducerMetrics(); - registerProducerTopicMetrics(); - registerConsumerMetrics(); - } + Map classNameToType = new HashMap<>(); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.Rate", + INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.Avg", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.Max", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.Value", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.CumulativeSum", + INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER); + classNameToType.put( + "org.apache.kafka.common.metrics.stats.CumulativeCount", + INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER); - private static void registerProducerMetrics() { - String group = "producer-metrics"; - Set tagKeys = Collections.singleton("client-id"); - - registry.put( - KafkaMetricId.create("outgoing-byte-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.outgoing-bytes.rate", - "The average number of outgoing bytes sent per second to all servers.", - unitBytesPerSecond)); - registry.put( - KafkaMetricId.create("response-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.responses.rate", - "The average number of responses received per second.", - "{responses}/s")); + for (Map.Entry entry : classNameToType.entrySet()) { + try { + measureableToInstrumentType.put(Class.forName(entry.getKey()), entry.getValue()); + } catch (ClassNotFoundException e) { + // Class doesn't exist in tis version of kafka client - skip + } + } } - private static void registerProducerTopicMetrics() { - String group = "producer-topic-metrics"; - Set tagKeys = new HashSet<>(Arrays.asList("client-id", "topic")); - - registry.put( - KafkaMetricId.create("byte-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.bytes.rate", - "The average number of bytes sent per second for a specific topic.", - unitBytesPerSecond)); - registry.put( - KafkaMetricId.create("compression-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.compression-ratio", - "The average compression ratio of record batches for a specific topic.", - "{compression}")); - registry.put( - KafkaMetricId.create("record-error-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.record-error.rate", - "The average per-second number of record sends that resulted in errors for a specific topic.", - "{errors}/s")); - registry.put( - KafkaMetricId.create("record-retry-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.record-retry.rate", - "The average per-second number of retried record sends for a specific topic.", - "{retries}/s")); - registry.put( - KafkaMetricId.create("record-send-rate", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.producer.record-sent.rate", - "The average number of records sent per second for a specific topic.", - "{records_sent}/s")); - } - - private static void registerConsumerMetrics() { - String group = "consumer-fetch-manager-metrics"; - Set tagKeys = new HashSet<>(Arrays.asList("client-id", "topic", "partition")); - - registry.put( - KafkaMetricId.create("records-lag", group, tagKeys), - MetricDescriptor.createDoubleGauge( - "messaging.kafka.consumer.lag", - "Current approximate lag of consumer group at partition of topic.", - "{lag}")); - } - - /** - * Returns the description of the OpenTelemetry metric definition for the kafka metric, or {@code - * null} if no mapping exists. - */ @Nullable - static MetricDescriptor getRegisteredInstrument(KafkaMetricId kafkaMetricId) { - return registry.get(kafkaMetricId); - } + static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) { + // If metric is not a Measureable, we can't map it to an instrument + if (kafkaMetricId.getMeasureable() == null) { + return null; + } + String instrumentName = kafkaMetricId.getGroup() + "." + kafkaMetricId.getName(); + String description = + descriptionCache.computeIfAbsent(instrumentName, s -> kafkaMetricId.getDescription()); + String instrumentType = + measureableToInstrumentType.entrySet().stream() + .filter(entry -> entry.getKey().equals(kafkaMetricId.getMeasureable())) + .findFirst() + .map(Map.Entry::getValue) + .orElse(INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); - // Visible for testing - static Set getMetricDescriptors() { - return new HashSet<>(registry.values()); + switch (instrumentType) { + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: + return InstrumentDescriptor.createDoubleGauge(instrumentName, description); + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: + return InstrumentDescriptor.createDoubleCounter(instrumentName, description); + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER: + return InstrumentDescriptor.createDoubleUpDownCounter(instrumentName, description); + default: // Continue below to throw + } + throw new IllegalStateException("Unrecognized instrument type. This is a bug."); } private KafkaMetricRegistry() {} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java deleted file mode 100644 index 44ca7b8eec5a..000000000000 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/MetricDescriptor.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.instrumentation.kafkaclients; - -import com.google.auto.value.AutoValue; - -/** A description of an OpenTelemetry metric instrument. */ -@AutoValue -abstract class MetricDescriptor { - - static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE = "DOUBLE_OBSERVABLE_GAUGE"; - - abstract String getName(); - - abstract String getDescription(); - - abstract String getUnit(); - - abstract String getInstrumentType(); - - static MetricDescriptor createDoubleGauge(String name, String description, String unit) { - return new AutoValue_MetricDescriptor( - name, description, unit, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); - } -} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index c657c0114605..fec1fb8bfd56 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -5,7 +5,11 @@ package io.opentelemetry.instrumentation.kafkaclients; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; +import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER; import static java.lang.System.lineSeparator; +import static java.util.Comparator.comparing; import static java.util.stream.Collectors.groupingBy; import static java.util.stream.Collectors.toList; @@ -13,13 +17,14 @@ import io.opentelemetry.api.common.Attributes; import io.opentelemetry.api.common.AttributesBuilder; import io.opentelemetry.api.metrics.Meter; -import java.util.Comparator; +import io.opentelemetry.api.metrics.ObservableDoubleMeasurement; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Optional; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; import java.util.logging.Level; import java.util.logging.Logger; import javax.annotation.Nullable; @@ -36,7 +41,7 @@ * // Map config = new HashMap<>(); * // // Register OpenTelemetryKafkaMetrics as reporter * // config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); - * // config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); + * // config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ...); * // ... * // try (KafkaProducer producer = new KafkaProducer<>(config)) { ... } * } @@ -84,35 +89,36 @@ static void printMappingTable() { StringBuilder sb = new StringBuilder(); // Append table headers sb.append( - "| Kafka Group | Kafka Name | Kafka Description | Attribute Keys | Instrument Name | Instrument Description | Instrument Unit | Instrument Type |") + "| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type |") .append(lineSeparator()) .append( - "|-------------|------------|-------------------|----------------|-----------------|------------------------|-----------------|-----------------|") + "|---|--------------|-------------|----------------|-----------------|------------------------|-----------------|") .append(lineSeparator()); Map> kafkaMetricsByGroup = seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup)); + int count = 1; // Iterate through groups in alpha order for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { List kafkaMetricIds = kafkaMetricsByGroup.get(group).stream() - .sorted(Comparator.comparing(KafkaMetricId::getName)) + .sorted(comparing(KafkaMetricId::getName)) .collect(toList()); // Iterate through metrics in alpha order by name for (KafkaMetricId kafkaMetricId : kafkaMetricIds) { - Optional descriptor = - Optional.ofNullable(KafkaMetricRegistry.getRegisteredInstrument(kafkaMetricId)); + Optional descriptor = + Optional.ofNullable(KafkaMetricRegistry.getInstrumentDescriptor(kafkaMetricId)); // Append table row sb.append( String.format( - "| %s | %s | %s | %s | %s | %s | %s | %s |%n", + "| %s | %s | %s | %s | %s | %s | %s |%n", + count, group, kafkaMetricId.getName(), - kafkaMetricId.getDescription(), - String.join(",", kafkaMetricId.getTagKeys()), - descriptor.map(MetricDescriptor::getName).orElse(""), - descriptor.map(MetricDescriptor::getDescription).orElse(""), - descriptor.map(MetricDescriptor::getUnit).orElse(""), - descriptor.map(MetricDescriptor::getInstrumentType).orElse(""))); + String.join(",", kafkaMetricId.getAttributeKeys()), + descriptor.map(InstrumentDescriptor::getName).orElse(""), + descriptor.map(InstrumentDescriptor::getDescription).orElse(""), + descriptor.map(InstrumentDescriptor::getInstrumentType).orElse(""))); + count++; } } logger.log(Level.INFO, "Mapping table" + System.lineSeparator() + sb); @@ -133,11 +139,10 @@ public void metricChange(KafkaMetric metric) { return; } - MetricDescriptor metricDescriptor = KafkaMetricRegistry.getRegisteredInstrument(kafkaMetricId); - if (metricDescriptor == null) { - logger.log( - Level.FINEST, - "Metric changed but did not match any metrics from registry: " + kafkaMetricId); + InstrumentDescriptor instrumentDescriptor = + KafkaMetricRegistry.getInstrumentDescriptor(kafkaMetricId); + if (instrumentDescriptor == null) { + logger.log(Level.FINEST, "Metric changed but cannot map to instrument: " + kafkaMetricId); return; } @@ -159,31 +164,43 @@ public void metricChange(KafkaMetric metric) { } else { logger.log(Level.FINEST, "Adding instrument " + registeredInstrument1); } - return createObservable(currentMeter, registeredInstrument1, metricDescriptor, metric); + return createObservable( + currentMeter, registeredInstrument1, instrumentDescriptor, metric); }); } private static AutoCloseable createObservable( Meter meter, RegisteredInstrument registeredInstrument, - MetricDescriptor metricDescriptor, + InstrumentDescriptor instrumentDescriptor, KafkaMetric kafkaMetric) { - if (metricDescriptor - .getInstrumentType() - .equals(MetricDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE)) { - return meter - .gaugeBuilder(metricDescriptor.getName()) - .setDescription(metricDescriptor.getDescription()) - .setUnit(metricDescriptor.getUnit()) - .buildWithCallback( - observableMeasurement -> - observableMeasurement.record( - kafkaMetric.value(), registeredInstrument.getAttributes())); + Consumer callback = + observableMeasurement -> + observableMeasurement.record(kafkaMetric.value(), registeredInstrument.getAttributes()); + switch (instrumentDescriptor.getInstrumentType()) { + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: + return meter + .gaugeBuilder(instrumentDescriptor.getName()) + .setDescription(instrumentDescriptor.getDescription()) + .buildWithCallback(callback); + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: + return meter + .counterBuilder(instrumentDescriptor.getName()) + .setDescription(instrumentDescriptor.getDescription()) + .ofDoubles() + .buildWithCallback(callback); + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER: + return meter + .upDownCounterBuilder(instrumentDescriptor.getName()) + .setDescription(instrumentDescriptor.getDescription()) + .ofDoubles() + .buildWithCallback(callback); + default: // Continue below to throw } // TODO: add support for other instrument types and value types as needed for new instruments // registered in KafkaMetricRegistry. // This should not happen. - throw new IllegalStateException("Unsupported metric descriptor: " + metricDescriptor); + throw new IllegalStateException("Unrecognized instrument type. This is a bug."); } private static Attributes toAttributes(KafkaMetric kafkaMetric) { From fe89d1cd8d28c5051d1689e2512fa937058c25a5 Mon Sep 17 00:00:00 2001 From: jack-berg Date: Fri, 10 Jun 2022 14:14:11 -0500 Subject: [PATCH 04/17] Include kafka prefix, fix typo --- .../OpenTelemetryKafkaMetricsTest.java | 376 +++++++++--------- .../kafkaclients/KafkaMetricRegistry.java | 14 +- 2 files changed, 199 insertions(+), 191 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index 8f5c59212ec9..d121f5161e9a 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -51,195 +51,193 @@ void observeMetrics() { Set expectedMetricNames = new HashSet<>( Arrays.asList( - "consumer-coordinator-metrics.assigned-partitions", - "consumer-coordinator-metrics.commit-latency-avg", - "consumer-coordinator-metrics.commit-latency-max", - "consumer-coordinator-metrics.commit-rate", - "consumer-coordinator-metrics.commit-total", - "consumer-coordinator-metrics.failed-rebalance-rate-per-hour", - "consumer-coordinator-metrics.failed-rebalance-total", - "consumer-coordinator-metrics.heartbeat-rate", - "consumer-coordinator-metrics.heartbeat-response-time-max", - "consumer-coordinator-metrics.heartbeat-total", - "consumer-coordinator-metrics.join-rate", - "consumer-coordinator-metrics.join-time-avg", - "consumer-coordinator-metrics.join-time-max", - "consumer-coordinator-metrics.join-total", - "consumer-coordinator-metrics.last-heartbeat-seconds-ago", - "consumer-coordinator-metrics.last-rebalance-seconds-ago", - "consumer-coordinator-metrics.partition-assigned-latency-avg", - "consumer-coordinator-metrics.partition-assigned-latency-max", - "consumer-coordinator-metrics.partition-lost-latency-avg", - "consumer-coordinator-metrics.partition-lost-latency-max", - "consumer-coordinator-metrics.partition-revoked-latency-avg", - "consumer-coordinator-metrics.partition-revoked-latency-max", - "consumer-coordinator-metrics.rebalance-latency-avg", - "consumer-coordinator-metrics.rebalance-latency-max", - "consumer-coordinator-metrics.rebalance-latency-total", - "consumer-coordinator-metrics.rebalance-rate-per-hour", - "consumer-coordinator-metrics.rebalance-total", - "consumer-coordinator-metrics.sync-rate", - "consumer-coordinator-metrics.sync-time-avg", - "consumer-coordinator-metrics.sync-time-max", - "consumer-coordinator-metrics.sync-total", - "consumer-fetch-manager-metrics.bytes-consumed-rate", - "consumer-fetch-manager-metrics.bytes-consumed-total", - "consumer-fetch-manager-metrics.fetch-latency-avg", - "consumer-fetch-manager-metrics.fetch-latency-max", - "consumer-fetch-manager-metrics.fetch-rate", - "consumer-fetch-manager-metrics.fetch-size-avg", - "consumer-fetch-manager-metrics.fetch-size-max", - "consumer-fetch-manager-metrics.fetch-throttle-time-avg", - "consumer-fetch-manager-metrics.fetch-throttle-time-max", - "consumer-fetch-manager-metrics.fetch-total", - "consumer-fetch-manager-metrics.records-consumed-rate", - "consumer-fetch-manager-metrics.records-consumed-total", - "consumer-fetch-manager-metrics.records-lag", - "consumer-fetch-manager-metrics.records-lag-avg", - "consumer-fetch-manager-metrics.records-lag-max", - "consumer-fetch-manager-metrics.records-lead", - "consumer-fetch-manager-metrics.records-lead-avg", - "consumer-fetch-manager-metrics.records-lead-min", - "consumer-fetch-manager-metrics.records-per-request-avg", - "consumer-metrics.connection-close-rate", - "consumer-metrics.connection-close-total", - "consumer-metrics.connection-count", - "consumer-metrics.connection-creation-rate", - "consumer-metrics.connection-creation-total", - "consumer-metrics.failed-authentication-rate", - "consumer-metrics.failed-authentication-total", - "consumer-metrics.failed-reauthentication-rate", - "consumer-metrics.failed-reauthentication-total", - "consumer-metrics.incoming-byte-rate", - "consumer-metrics.incoming-byte-total", - "consumer-metrics.io-ratio", - "consumer-metrics.io-time-ns-avg", - "consumer-metrics.io-wait-ratio", - "consumer-metrics.io-wait-time-ns-avg", - "consumer-metrics.io-waittime-total", - "consumer-metrics.iotime-total", - "consumer-metrics.last-poll-seconds-ago", - "consumer-metrics.network-io-rate", - "consumer-metrics.network-io-total", - "consumer-metrics.outgoing-byte-rate", - "consumer-metrics.outgoing-byte-total", - "consumer-metrics.poll-idle-ratio-avg", - "consumer-metrics.reauthentication-latency-avg", - "consumer-metrics.reauthentication-latency-max", - "consumer-metrics.request-rate", - "consumer-metrics.request-size-avg", - "consumer-metrics.request-size-max", - "consumer-metrics.request-total", - "consumer-metrics.response-rate", - "consumer-metrics.response-total", - "consumer-metrics.select-rate", - "consumer-metrics.select-total", - "consumer-metrics.successful-authentication-no-reauth-total", - "consumer-metrics.successful-authentication-rate", - "consumer-metrics.successful-authentication-total", - "consumer-metrics.successful-reauthentication-rate", - "consumer-metrics.successful-reauthentication-total", - "consumer-metrics.time-between-poll-avg", - "consumer-metrics.time-between-poll-max", - "consumer-node-metrics.incoming-byte-rate", - "consumer-node-metrics.incoming-byte-total", - "consumer-node-metrics.outgoing-byte-rate", - "consumer-node-metrics.outgoing-byte-total", - "consumer-node-metrics.request-latency-avg", - "consumer-node-metrics.request-latency-max", - "consumer-node-metrics.request-rate", - "consumer-node-metrics.request-size-avg", - "consumer-node-metrics.request-size-max", - "consumer-node-metrics.request-total", - "consumer-node-metrics.response-rate", - "consumer-node-metrics.response-total", - "kafka-metrics-count.count", - "producer-metrics.batch-size-avg", - "producer-metrics.batch-size-max", - "producer-metrics.batch-split-rate", - "producer-metrics.batch-split-total", - "producer-metrics.buffer-available-bytes", - "producer-metrics.buffer-exhausted-rate", - "producer-metrics.buffer-exhausted-total", - "producer-metrics.buffer-total-bytes", - "producer-metrics.bufferpool-wait-ratio", - "producer-metrics.bufferpool-wait-time-total", - "producer-metrics.compression-rate-avg", - "producer-metrics.connection-close-rate", - "producer-metrics.connection-close-total", - "producer-metrics.connection-count", - "producer-metrics.connection-creation-rate", - "producer-metrics.connection-creation-total", - "producer-metrics.failed-authentication-rate", - "producer-metrics.failed-authentication-total", - "producer-metrics.failed-reauthentication-rate", - "producer-metrics.failed-reauthentication-total", - "producer-metrics.incoming-byte-rate", - "producer-metrics.incoming-byte-total", - "producer-metrics.io-ratio", - "producer-metrics.io-time-ns-avg", - "producer-metrics.io-wait-ratio", - "producer-metrics.io-wait-time-ns-avg", - "producer-metrics.io-waittime-total", - "producer-metrics.iotime-total", - "producer-metrics.metadata-age", - "producer-metrics.network-io-rate", - "producer-metrics.network-io-total", - "producer-metrics.outgoing-byte-rate", - "producer-metrics.outgoing-byte-total", - "producer-metrics.produce-throttle-time-avg", - "producer-metrics.produce-throttle-time-max", - "producer-metrics.reauthentication-latency-avg", - "producer-metrics.reauthentication-latency-max", - "producer-metrics.record-error-rate", - "producer-metrics.record-error-total", - "producer-metrics.record-queue-time-avg", - "producer-metrics.record-queue-time-max", - "producer-metrics.record-retry-rate", - "producer-metrics.record-retry-total", - "producer-metrics.record-send-rate", - "producer-metrics.record-send-total", - "producer-metrics.record-size-avg", - "producer-metrics.record-size-max", - "producer-metrics.records-per-request-avg", - "producer-metrics.request-latency-avg", - "producer-metrics.request-latency-max", - "producer-metrics.request-rate", - "producer-metrics.request-size-avg", - "producer-metrics.request-size-max", - "producer-metrics.request-total", - "producer-metrics.requests-in-flight", - "producer-metrics.response-rate", - "producer-metrics.response-total", - "producer-metrics.select-rate", - "producer-metrics.select-total", - "producer-metrics.successful-authentication-no-reauth-total", - "producer-metrics.successful-authentication-rate", - "producer-metrics.successful-authentication-total", - "producer-metrics.successful-reauthentication-rate", - "producer-metrics.successful-reauthentication-total", - "producer-metrics.waiting-threads", - "producer-node-metrics.incoming-byte-rate", - "producer-node-metrics.incoming-byte-total", - "producer-node-metrics.outgoing-byte-rate", - "producer-node-metrics.outgoing-byte-total", - "producer-node-metrics.request-latency-avg", - "producer-node-metrics.request-latency-max", - "producer-node-metrics.request-rate", - "producer-node-metrics.request-size-avg", - "producer-node-metrics.request-size-max", - "producer-node-metrics.request-total", - "producer-node-metrics.response-rate", - "producer-node-metrics.response-total", - "producer-topic-metrics.byte-rate", - "producer-topic-metrics.byte-total", - "producer-topic-metrics.compression-rate", - "producer-topic-metrics.record-error-rate", - "producer-topic-metrics.record-error-total", - "producer-topic-metrics.record-retry-rate", - "producer-topic-metrics.record-retry-total", - "producer-topic-metrics.record-send-rate", - "producer-topic-metrics.record-send-total")); + "kafka.consumer.commit-latency-avg", + "kafka.consumer.commit-latency-max", + "kafka.consumer.commit-rate", + "kafka.consumer.commit-total", + "kafka.consumer.failed-rebalance-rate-per-hour", + "kafka.consumer.failed-rebalance-total", + "kafka.consumer.heartbeat-rate", + "kafka.consumer.heartbeat-response-time-max", + "kafka.consumer.heartbeat-total", + "kafka.consumer.join-rate", + "kafka.consumer.join-time-avg", + "kafka.consumer.join-time-max", + "kafka.consumer.join-total", + "kafka.consumer.last-heartbeat-seconds-ago", + "kafka.consumer.last-rebalance-seconds-ago", + "kafka.consumer.partition-assigned-latency-avg", + "kafka.consumer.partition-assigned-latency-max", + "kafka.consumer.partition-lost-latency-avg", + "kafka.consumer.partition-lost-latency-max", + "kafka.consumer.partition-revoked-latency-avg", + "kafka.consumer.partition-revoked-latency-max", + "kafka.consumer.rebalance-latency-avg", + "kafka.consumer.rebalance-latency-max", + "kafka.consumer.rebalance-latency-total", + "kafka.consumer.rebalance-rate-per-hour", + "kafka.consumer.rebalance-total", + "kafka.consumer.sync-rate", + "kafka.consumer.sync-time-avg", + "kafka.consumer.sync-time-max", + "kafka.consumer.sync-total", + "kafka.consumer.bytes-consumed-rate", + "kafka.consumer.bytes-consumed-total", + "kafka.consumer.fetch-latency-avg", + "kafka.consumer.fetch-latency-max", + "kafka.consumer.fetch-rate", + "kafka.consumer.fetch-size-avg", + "kafka.consumer.fetch-size-max", + "kafka.consumer.fetch-throttle-time-avg", + "kafka.consumer.fetch-throttle-time-max", + "kafka.consumer.fetch-total", + "kafka.consumer.records-consumed-rate", + "kafka.consumer.records-consumed-total", + "kafka.consumer.records-lag", + "kafka.consumer.records-lag-avg", + "kafka.consumer.records-lag-max", + "kafka.consumer.records-lead", + "kafka.consumer.records-lead-avg", + "kafka.consumer.records-lead-min", + "kafka.consumer.records-per-request-avg", + "kafka.consumer.connection-close-rate", + "kafka.consumer.connection-close-total", + "kafka.consumer.connection-count", + "kafka.consumer.connection-creation-rate", + "kafka.consumer.connection-creation-total", + "kafka.consumer.failed-authentication-rate", + "kafka.consumer.failed-authentication-total", + "kafka.consumer.failed-reauthentication-rate", + "kafka.consumer.failed-reauthentication-total", + "kafka.consumer.incoming-byte-rate", + "kafka.consumer.incoming-byte-total", + "kafka.consumer.io-ratio", + "kafka.consumer.io-time-ns-avg", + "kafka.consumer.io-wait-ratio", + "kafka.consumer.io-wait-time-ns-avg", + "kafka.consumer.io-waittime-total", + "kafka.consumer.iotime-total", + "kafka.consumer.last-poll-seconds-ago", + "kafka.consumer.network-io-rate", + "kafka.consumer.network-io-total", + "kafka.consumer.outgoing-byte-rate", + "kafka.consumer.outgoing-byte-total", + "kafka.consumer.poll-idle-ratio-avg", + "kafka.consumer.reauthentication-latency-avg", + "kafka.consumer.reauthentication-latency-max", + "kafka.consumer.request-rate", + "kafka.consumer.request-size-avg", + "kafka.consumer.request-size-max", + "kafka.consumer.request-total", + "kafka.consumer.response-rate", + "kafka.consumer.response-total", + "kafka.consumer.select-rate", + "kafka.consumer.select-total", + "kafka.consumer.successful-authentication-no-reauth-total", + "kafka.consumer.successful-authentication-rate", + "kafka.consumer.successful-authentication-total", + "kafka.consumer.successful-reauthentication-rate", + "kafka.consumer.successful-reauthentication-total", + "kafka.consumer.time-between-poll-avg", + "kafka.consumer.time-between-poll-max", + "kafka.consumer.incoming-byte-rate", + "kafka.consumer.incoming-byte-total", + "kafka.consumer.outgoing-byte-rate", + "kafka.consumer.outgoing-byte-total", + "kafka.consumer.request-latency-avg", + "kafka.consumer.request-latency-max", + "kafka.consumer.request-rate", + "kafka.consumer.request-size-avg", + "kafka.consumer.request-size-max", + "kafka.consumer.request-total", + "kafka.consumer.response-rate", + "kafka.consumer.response-total", + "kafka.producer.batch-size-avg", + "kafka.producer.batch-size-max", + "kafka.producer.batch-split-rate", + "kafka.producer.batch-split-total", + "kafka.producer.buffer-available-bytes", + "kafka.producer.buffer-exhausted-rate", + "kafka.producer.buffer-exhausted-total", + "kafka.producer.buffer-total-bytes", + "kafka.producer.bufferpool-wait-ratio", + "kafka.producer.bufferpool-wait-time-total", + "kafka.producer.compression-rate-avg", + "kafka.producer.connection-close-rate", + "kafka.producer.connection-close-total", + "kafka.producer.connection-count", + "kafka.producer.connection-creation-rate", + "kafka.producer.connection-creation-total", + "kafka.producer.failed-authentication-rate", + "kafka.producer.failed-authentication-total", + "kafka.producer.failed-reauthentication-rate", + "kafka.producer.failed-reauthentication-total", + "kafka.producer.incoming-byte-rate", + "kafka.producer.incoming-byte-total", + "kafka.producer.io-ratio", + "kafka.producer.io-time-ns-avg", + "kafka.producer.io-wait-ratio", + "kafka.producer.io-wait-time-ns-avg", + "kafka.producer.io-waittime-total", + "kafka.producer.iotime-total", + "kafka.producer.metadata-age", + "kafka.producer.network-io-rate", + "kafka.producer.network-io-total", + "kafka.producer.outgoing-byte-rate", + "kafka.producer.outgoing-byte-total", + "kafka.producer.produce-throttle-time-avg", + "kafka.producer.produce-throttle-time-max", + "kafka.producer.reauthentication-latency-avg", + "kafka.producer.reauthentication-latency-max", + "kafka.producer.record-error-rate", + "kafka.producer.record-error-total", + "kafka.producer.record-queue-time-avg", + "kafka.producer.record-queue-time-max", + "kafka.producer.record-retry-rate", + "kafka.producer.record-retry-total", + "kafka.producer.record-send-rate", + "kafka.producer.record-send-total", + "kafka.producer.record-size-avg", + "kafka.producer.record-size-max", + "kafka.producer.records-per-request-avg", + "kafka.producer.request-latency-avg", + "kafka.producer.request-latency-max", + "kafka.producer.request-rate", + "kafka.producer.request-size-avg", + "kafka.producer.request-size-max", + "kafka.producer.request-total", + "kafka.producer.requests-in-flight", + "kafka.producer.response-rate", + "kafka.producer.response-total", + "kafka.producer.select-rate", + "kafka.producer.select-total", + "kafka.producer.successful-authentication-no-reauth-total", + "kafka.producer.successful-authentication-rate", + "kafka.producer.successful-authentication-total", + "kafka.producer.successful-reauthentication-rate", + "kafka.producer.successful-reauthentication-total", + "kafka.producer.waiting-threads", + "kafka.producer.incoming-byte-rate", + "kafka.producer.incoming-byte-total", + "kafka.producer.outgoing-byte-rate", + "kafka.producer.outgoing-byte-total", + "kafka.producer.request-latency-avg", + "kafka.producer.request-latency-max", + "kafka.producer.request-rate", + "kafka.producer.request-size-avg", + "kafka.producer.request-size-max", + "kafka.producer.request-total", + "kafka.producer.response-rate", + "kafka.producer.response-total", + "kafka.producer.byte-rate", + "kafka.producer.byte-total", + "kafka.producer.compression-rate", + "kafka.producer.record-error-rate", + "kafka.producer.record-error-total", + "kafka.producer.record-retry-rate", + "kafka.producer.record-retry-total", + "kafka.producer.record-send-rate", + "kafka.producer.record-send-total")); Set metricNames = testing.metrics().stream().map(MetricData::getName).collect(toSet()); diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index 4911921f297e..cda51cf800c7 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -9,14 +9,19 @@ import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER; +import java.util.Arrays; import java.util.HashMap; +import java.util.HashSet; import java.util.Map; +import java.util.Optional; +import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import javax.annotation.Nullable; /** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */ class KafkaMetricRegistry { + private static final Set groups = new HashSet<>(Arrays.asList("consumer", "producer")); private static final Map, String> measureableToInstrumentType = new HashMap<>(); private static final Map descriptionCache = new ConcurrentHashMap<>(); @@ -42,7 +47,7 @@ class KafkaMetricRegistry { try { measureableToInstrumentType.put(Class.forName(entry.getKey()), entry.getValue()); } catch (ClassNotFoundException e) { - // Class doesn't exist in tis version of kafka client - skip + // Class doesn't exist in this version of kafka client - skip } } } @@ -53,7 +58,12 @@ static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) if (kafkaMetricId.getMeasureable() == null) { return null; } - String instrumentName = kafkaMetricId.getGroup() + "." + kafkaMetricId.getName(); + Optional matchingGroup = groups.stream().filter(group -> kafkaMetricId.getGroup().contains(group)).findFirst(); + // Only map metrics that have a matching group + if (!matchingGroup.isPresent()) { + return null; + } + String instrumentName = "kafka." + matchingGroup.get() + "." + kafkaMetricId.getName(); String description = descriptionCache.computeIfAbsent(instrumentName, s -> kafkaMetricId.getDescription()); String instrumentType = From ef866fce7d79980ae41a8690f8257bf8d22ee986 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Mon, 13 Jun 2022 18:08:22 -0500 Subject: [PATCH 05/17] Spotless, update readme with latest metric names --- instrumentation/kafka/kafka-clients/README.md | 396 +++++++++--------- .../kafkaclients/KafkaMetricRegistry.java | 3 +- 2 files changed, 200 insertions(+), 199 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/README.md b/instrumentation/kafka/kafka-clients/README.md index 8ed4ea06f18e..4929ac092e77 100644 --- a/instrumentation/kafka/kafka-clients/README.md +++ b/instrumentation/kafka/kafka-clients/README.md @@ -32,202 +32,202 @@ collected. | 1 | app-info | commit-id | client-id | | | | | 2 | app-info | start-time-ms | client-id | | | | | 3 | app-info | version | client-id | | | | -| 4 | consumer-coordinator-metrics | assigned-partitions | client-id | consumer-coordinator-metrics.assigned-partitions | The number of partitions currently assigned to this consumer | DOUBLE_OBSERVABLE_GAUGE | -| 5 | consumer-coordinator-metrics | commit-latency-avg | client-id | consumer-coordinator-metrics.commit-latency-avg | The average time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | -| 6 | consumer-coordinator-metrics | commit-latency-max | client-id | consumer-coordinator-metrics.commit-latency-max | The max time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | -| 7 | consumer-coordinator-metrics | commit-rate | client-id | consumer-coordinator-metrics.commit-rate | The number of commit calls per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 8 | consumer-coordinator-metrics | commit-total | client-id | consumer-coordinator-metrics.commit-total | The total number of commit calls | DOUBLE_OBSERVABLE_COUNTER | -| 9 | consumer-coordinator-metrics | failed-rebalance-rate-per-hour | client-id | consumer-coordinator-metrics.failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 10 | consumer-coordinator-metrics | failed-rebalance-total | client-id | consumer-coordinator-metrics.failed-rebalance-total | The total number of failed rebalance events | DOUBLE_OBSERVABLE_COUNTER | -| 11 | consumer-coordinator-metrics | heartbeat-rate | client-id | consumer-coordinator-metrics.heartbeat-rate | The number of heartbeats per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 12 | consumer-coordinator-metrics | heartbeat-response-time-max | client-id | consumer-coordinator-metrics.heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | DOUBLE_OBSERVABLE_GAUGE | -| 13 | consumer-coordinator-metrics | heartbeat-total | client-id | consumer-coordinator-metrics.heartbeat-total | The total number of heartbeats | DOUBLE_OBSERVABLE_COUNTER | -| 14 | consumer-coordinator-metrics | join-rate | client-id | consumer-coordinator-metrics.join-rate | The number of group joins per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 15 | consumer-coordinator-metrics | join-time-avg | client-id | consumer-coordinator-metrics.join-time-avg | The average time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | -| 16 | consumer-coordinator-metrics | join-time-max | client-id | consumer-coordinator-metrics.join-time-max | The max time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | -| 17 | consumer-coordinator-metrics | join-total | client-id | consumer-coordinator-metrics.join-total | The total number of group joins | DOUBLE_OBSERVABLE_COUNTER | -| 18 | consumer-coordinator-metrics | last-heartbeat-seconds-ago | client-id | consumer-coordinator-metrics.last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | DOUBLE_OBSERVABLE_GAUGE | -| 19 | consumer-coordinator-metrics | last-rebalance-seconds-ago | client-id | consumer-coordinator-metrics.last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | DOUBLE_OBSERVABLE_GAUGE | -| 20 | consumer-coordinator-metrics | partition-assigned-latency-avg | client-id | consumer-coordinator-metrics.partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 21 | consumer-coordinator-metrics | partition-assigned-latency-max | client-id | consumer-coordinator-metrics.partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 22 | consumer-coordinator-metrics | partition-lost-latency-avg | client-id | consumer-coordinator-metrics.partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 23 | consumer-coordinator-metrics | partition-lost-latency-max | client-id | consumer-coordinator-metrics.partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 24 | consumer-coordinator-metrics | partition-revoked-latency-avg | client-id | consumer-coordinator-metrics.partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 25 | consumer-coordinator-metrics | partition-revoked-latency-max | client-id | consumer-coordinator-metrics.partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 26 | consumer-coordinator-metrics | rebalance-latency-avg | client-id | consumer-coordinator-metrics.rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | -| 27 | consumer-coordinator-metrics | rebalance-latency-max | client-id | consumer-coordinator-metrics.rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | -| 28 | consumer-coordinator-metrics | rebalance-latency-total | client-id | consumer-coordinator-metrics.rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | DOUBLE_OBSERVABLE_COUNTER | -| 29 | consumer-coordinator-metrics | rebalance-rate-per-hour | client-id | consumer-coordinator-metrics.rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 30 | consumer-coordinator-metrics | rebalance-total | client-id | consumer-coordinator-metrics.rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_COUNTER | -| 31 | consumer-coordinator-metrics | sync-rate | client-id | consumer-coordinator-metrics.sync-rate | The number of group syncs per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 32 | consumer-coordinator-metrics | sync-time-avg | client-id | consumer-coordinator-metrics.sync-time-avg | The average time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | -| 33 | consumer-coordinator-metrics | sync-time-max | client-id | consumer-coordinator-metrics.sync-time-max | The max time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | -| 34 | consumer-coordinator-metrics | sync-total | client-id | consumer-coordinator-metrics.sync-total | The total number of group syncs | DOUBLE_OBSERVABLE_COUNTER | -| 35 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id | consumer-fetch-manager-metrics.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 36 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id,topic | consumer-fetch-manager-metrics.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 37 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id | consumer-fetch-manager-metrics.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | -| 38 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id,topic | consumer-fetch-manager-metrics.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | -| 39 | consumer-fetch-manager-metrics | fetch-latency-avg | client-id | consumer-fetch-manager-metrics.fetch-latency-avg | The average time taken for a fetch request. | DOUBLE_OBSERVABLE_GAUGE | -| 40 | consumer-fetch-manager-metrics | fetch-latency-max | client-id | consumer-fetch-manager-metrics.fetch-latency-max | The max time taken for any fetch request. | DOUBLE_OBSERVABLE_GAUGE | -| 41 | consumer-fetch-manager-metrics | fetch-rate | client-id | consumer-fetch-manager-metrics.fetch-rate | The number of fetch requests per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 42 | consumer-fetch-manager-metrics | fetch-size-avg | client-id | consumer-fetch-manager-metrics.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 43 | consumer-fetch-manager-metrics | fetch-size-avg | client-id,topic | consumer-fetch-manager-metrics.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 44 | consumer-fetch-manager-metrics | fetch-size-max | client-id | consumer-fetch-manager-metrics.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 45 | consumer-fetch-manager-metrics | fetch-size-max | client-id,topic | consumer-fetch-manager-metrics.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 46 | consumer-fetch-manager-metrics | fetch-throttle-time-avg | client-id | consumer-fetch-manager-metrics.fetch-throttle-time-avg | The average throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | -| 47 | consumer-fetch-manager-metrics | fetch-throttle-time-max | client-id | consumer-fetch-manager-metrics.fetch-throttle-time-max | The maximum throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | -| 48 | consumer-fetch-manager-metrics | fetch-total | client-id | consumer-fetch-manager-metrics.fetch-total | The total number of fetch requests. | DOUBLE_OBSERVABLE_COUNTER | +| 4 | consumer-coordinator-metrics | assigned-partitions | client-id | kafka.consumer.assigned-partitions | The number of partitions currently assigned to this consumer | DOUBLE_OBSERVABLE_GAUGE | +| 5 | consumer-coordinator-metrics | commit-latency-avg | client-id | kafka.consumer.commit-latency-avg | The average time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | +| 6 | consumer-coordinator-metrics | commit-latency-max | client-id | kafka.consumer.commit-latency-max | The max time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | +| 7 | consumer-coordinator-metrics | commit-rate | client-id | kafka.consumer.commit-rate | The number of commit calls per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 8 | consumer-coordinator-metrics | commit-total | client-id | kafka.consumer.commit-total | The total number of commit calls | DOUBLE_OBSERVABLE_COUNTER | +| 9 | consumer-coordinator-metrics | failed-rebalance-rate-per-hour | client-id | kafka.consumer.failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 10 | consumer-coordinator-metrics | failed-rebalance-total | client-id | kafka.consumer.failed-rebalance-total | The total number of failed rebalance events | DOUBLE_OBSERVABLE_COUNTER | +| 11 | consumer-coordinator-metrics | heartbeat-rate | client-id | kafka.consumer.heartbeat-rate | The number of heartbeats per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 12 | consumer-coordinator-metrics | heartbeat-response-time-max | client-id | kafka.consumer.heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | DOUBLE_OBSERVABLE_GAUGE | +| 13 | consumer-coordinator-metrics | heartbeat-total | client-id | kafka.consumer.heartbeat-total | The total number of heartbeats | DOUBLE_OBSERVABLE_COUNTER | +| 14 | consumer-coordinator-metrics | join-rate | client-id | kafka.consumer.join-rate | The number of group joins per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 15 | consumer-coordinator-metrics | join-time-avg | client-id | kafka.consumer.join-time-avg | The average time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | +| 16 | consumer-coordinator-metrics | join-time-max | client-id | kafka.consumer.join-time-max | The max time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | +| 17 | consumer-coordinator-metrics | join-total | client-id | kafka.consumer.join-total | The total number of group joins | DOUBLE_OBSERVABLE_COUNTER | +| 18 | consumer-coordinator-metrics | last-heartbeat-seconds-ago | client-id | kafka.consumer.last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | DOUBLE_OBSERVABLE_GAUGE | +| 19 | consumer-coordinator-metrics | last-rebalance-seconds-ago | client-id | kafka.consumer.last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | DOUBLE_OBSERVABLE_GAUGE | +| 20 | consumer-coordinator-metrics | partition-assigned-latency-avg | client-id | kafka.consumer.partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 21 | consumer-coordinator-metrics | partition-assigned-latency-max | client-id | kafka.consumer.partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 22 | consumer-coordinator-metrics | partition-lost-latency-avg | client-id | kafka.consumer.partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 23 | consumer-coordinator-metrics | partition-lost-latency-max | client-id | kafka.consumer.partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 24 | consumer-coordinator-metrics | partition-revoked-latency-avg | client-id | kafka.consumer.partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 25 | consumer-coordinator-metrics | partition-revoked-latency-max | client-id | kafka.consumer.partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | +| 26 | consumer-coordinator-metrics | rebalance-latency-avg | client-id | kafka.consumer.rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | +| 27 | consumer-coordinator-metrics | rebalance-latency-max | client-id | kafka.consumer.rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | +| 28 | consumer-coordinator-metrics | rebalance-latency-total | client-id | kafka.consumer.rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | DOUBLE_OBSERVABLE_COUNTER | +| 29 | consumer-coordinator-metrics | rebalance-rate-per-hour | client-id | kafka.consumer.rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 30 | consumer-coordinator-metrics | rebalance-total | client-id | kafka.consumer.rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_COUNTER | +| 31 | consumer-coordinator-metrics | sync-rate | client-id | kafka.consumer.sync-rate | The number of group syncs per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 32 | consumer-coordinator-metrics | sync-time-avg | client-id | kafka.consumer.sync-time-avg | The average time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | +| 33 | consumer-coordinator-metrics | sync-time-max | client-id | kafka.consumer.sync-time-max | The max time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | +| 34 | consumer-coordinator-metrics | sync-total | client-id | kafka.consumer.sync-total | The total number of group syncs | DOUBLE_OBSERVABLE_COUNTER | +| 35 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id | kafka.consumer.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 36 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id,topic | kafka.consumer.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 37 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id,topic | kafka.consumer.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | +| 38 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id | kafka.consumer.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | +| 39 | consumer-fetch-manager-metrics | fetch-latency-avg | client-id | kafka.consumer.fetch-latency-avg | The average time taken for a fetch request. | DOUBLE_OBSERVABLE_GAUGE | +| 40 | consumer-fetch-manager-metrics | fetch-latency-max | client-id | kafka.consumer.fetch-latency-max | The max time taken for any fetch request. | DOUBLE_OBSERVABLE_GAUGE | +| 41 | consumer-fetch-manager-metrics | fetch-rate | client-id | kafka.consumer.fetch-rate | The number of fetch requests per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 42 | consumer-fetch-manager-metrics | fetch-size-avg | client-id,topic | kafka.consumer.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 43 | consumer-fetch-manager-metrics | fetch-size-avg | client-id | kafka.consumer.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 44 | consumer-fetch-manager-metrics | fetch-size-max | client-id,topic | kafka.consumer.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 45 | consumer-fetch-manager-metrics | fetch-size-max | client-id | kafka.consumer.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | +| 46 | consumer-fetch-manager-metrics | fetch-throttle-time-avg | client-id | kafka.consumer.fetch-throttle-time-avg | The average throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | +| 47 | consumer-fetch-manager-metrics | fetch-throttle-time-max | client-id | kafka.consumer.fetch-throttle-time-max | The maximum throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | +| 48 | consumer-fetch-manager-metrics | fetch-total | client-id | kafka.consumer.fetch-total | The total number of fetch requests. | DOUBLE_OBSERVABLE_COUNTER | | 49 | consumer-fetch-manager-metrics | preferred-read-replica | client-id,topic,partition | | | | -| 50 | consumer-fetch-manager-metrics | records-consumed-rate | client-id | consumer-fetch-manager-metrics.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 51 | consumer-fetch-manager-metrics | records-consumed-rate | client-id,topic | consumer-fetch-manager-metrics.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 52 | consumer-fetch-manager-metrics | records-consumed-total | client-id,topic | consumer-fetch-manager-metrics.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | -| 53 | consumer-fetch-manager-metrics | records-consumed-total | client-id | consumer-fetch-manager-metrics.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | -| 54 | consumer-fetch-manager-metrics | records-lag | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag | The latest lag of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 55 | consumer-fetch-manager-metrics | records-lag-avg | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag-avg | The average lag of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 56 | consumer-fetch-manager-metrics | records-lag-max | client-id,topic,partition | consumer-fetch-manager-metrics.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 57 | consumer-fetch-manager-metrics | records-lag-max | client-id | consumer-fetch-manager-metrics.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 58 | consumer-fetch-manager-metrics | records-lead | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead | The latest lead of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 59 | consumer-fetch-manager-metrics | records-lead-avg | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead-avg | The average lead of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 60 | consumer-fetch-manager-metrics | records-lead-min | client-id,topic,partition | consumer-fetch-manager-metrics.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 61 | consumer-fetch-manager-metrics | records-lead-min | client-id | consumer-fetch-manager-metrics.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 62 | consumer-fetch-manager-metrics | records-per-request-avg | client-id,topic | consumer-fetch-manager-metrics.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | -| 63 | consumer-fetch-manager-metrics | records-per-request-avg | client-id | consumer-fetch-manager-metrics.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | -| 64 | consumer-metrics | connection-close-rate | client-id | consumer-metrics.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 65 | consumer-metrics | connection-close-total | client-id | consumer-metrics.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | -| 66 | consumer-metrics | connection-count | client-id | consumer-metrics.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | -| 67 | consumer-metrics | connection-creation-rate | client-id | consumer-metrics.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 68 | consumer-metrics | connection-creation-total | client-id | consumer-metrics.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | -| 69 | consumer-metrics | failed-authentication-rate | client-id | consumer-metrics.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 70 | consumer-metrics | failed-authentication-total | client-id | consumer-metrics.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | -| 71 | consumer-metrics | failed-reauthentication-rate | client-id | consumer-metrics.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 72 | consumer-metrics | failed-reauthentication-total | client-id | consumer-metrics.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 73 | consumer-metrics | incoming-byte-rate | client-id | consumer-metrics.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 74 | consumer-metrics | incoming-byte-total | client-id | consumer-metrics.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 75 | consumer-metrics | io-ratio | client-id | consumer-metrics.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 76 | consumer-metrics | io-time-ns-avg | client-id | consumer-metrics.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 77 | consumer-metrics | io-wait-ratio | client-id | consumer-metrics.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 78 | consumer-metrics | io-wait-time-ns-avg | client-id | consumer-metrics.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 79 | consumer-metrics | io-waittime-total | client-id | consumer-metrics.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | -| 80 | consumer-metrics | iotime-total | client-id | consumer-metrics.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | -| 81 | consumer-metrics | last-poll-seconds-ago | client-id | consumer-metrics.last-poll-seconds-ago | The number of seconds since the last poll() invocation. | DOUBLE_OBSERVABLE_GAUGE | -| 82 | consumer-metrics | network-io-rate | client-id | consumer-metrics.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 83 | consumer-metrics | network-io-total | client-id | consumer-metrics.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | -| 84 | consumer-metrics | outgoing-byte-rate | client-id | consumer-metrics.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 85 | consumer-metrics | outgoing-byte-total | client-id | consumer-metrics.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 86 | consumer-metrics | poll-idle-ratio-avg | client-id | consumer-metrics.poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | DOUBLE_OBSERVABLE_GAUGE | -| 87 | consumer-metrics | reauthentication-latency-avg | client-id | consumer-metrics.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 88 | consumer-metrics | reauthentication-latency-max | client-id | consumer-metrics.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 89 | consumer-metrics | request-rate | client-id | consumer-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 90 | consumer-metrics | request-size-avg | client-id | consumer-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 91 | consumer-metrics | request-size-max | client-id | consumer-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 92 | consumer-metrics | request-total | client-id | consumer-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 93 | consumer-metrics | response-rate | client-id | consumer-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 94 | consumer-metrics | response-total | client-id | consumer-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 95 | consumer-metrics | select-rate | client-id | consumer-metrics.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 96 | consumer-metrics | select-total | client-id | consumer-metrics.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | -| 97 | consumer-metrics | successful-authentication-no-reauth-total | client-id | consumer-metrics.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | -| 98 | consumer-metrics | successful-authentication-rate | client-id | consumer-metrics.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 99 | consumer-metrics | successful-authentication-total | client-id | consumer-metrics.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | -| 100 | consumer-metrics | successful-reauthentication-rate | client-id | consumer-metrics.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 101 | consumer-metrics | successful-reauthentication-total | client-id | consumer-metrics.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 102 | consumer-metrics | time-between-poll-avg | client-id | consumer-metrics.time-between-poll-avg | The average delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | -| 103 | consumer-metrics | time-between-poll-max | client-id | consumer-metrics.time-between-poll-max | The max delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | -| 104 | consumer-node-metrics | incoming-byte-rate | client-id,node-id | consumer-node-metrics.incoming-byte-rate | The number of incoming bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 105 | consumer-node-metrics | incoming-byte-total | client-id,node-id | consumer-node-metrics.incoming-byte-total | The total number of incoming bytes | DOUBLE_OBSERVABLE_COUNTER | -| 106 | consumer-node-metrics | outgoing-byte-rate | client-id,node-id | consumer-node-metrics.outgoing-byte-rate | The number of outgoing bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 107 | consumer-node-metrics | outgoing-byte-total | client-id,node-id | consumer-node-metrics.outgoing-byte-total | The total number of outgoing bytes | DOUBLE_OBSERVABLE_COUNTER | -| 108 | consumer-node-metrics | request-latency-avg | client-id,node-id | consumer-node-metrics.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | -| 109 | consumer-node-metrics | request-latency-max | client-id,node-id | consumer-node-metrics.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | -| 110 | consumer-node-metrics | request-rate | client-id,node-id | consumer-node-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 111 | consumer-node-metrics | request-size-avg | client-id,node-id | consumer-node-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 112 | consumer-node-metrics | request-size-max | client-id,node-id | consumer-node-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 113 | consumer-node-metrics | request-total | client-id,node-id | consumer-node-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 114 | consumer-node-metrics | response-rate | client-id,node-id | consumer-node-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 115 | consumer-node-metrics | response-total | client-id,node-id | consumer-node-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 116 | kafka-metrics-count | count | client-id | kafka-metrics-count.count | total number of registered metrics | DOUBLE_OBSERVABLE_GAUGE | -| 117 | producer-metrics | batch-size-avg | client-id | producer-metrics.batch-size-avg | The average number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | -| 118 | producer-metrics | batch-size-max | client-id | producer-metrics.batch-size-max | The max number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | -| 119 | producer-metrics | batch-split-rate | client-id | producer-metrics.batch-split-rate | The average number of batch splits per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 120 | producer-metrics | batch-split-total | client-id | producer-metrics.batch-split-total | The total number of batch splits | DOUBLE_OBSERVABLE_COUNTER | -| 121 | producer-metrics | buffer-available-bytes | client-id | producer-metrics.buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | DOUBLE_OBSERVABLE_GAUGE | -| 122 | producer-metrics | buffer-exhausted-rate | client-id | producer-metrics.buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 123 | producer-metrics | buffer-exhausted-total | client-id | producer-metrics.buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_COUNTER | -| 124 | producer-metrics | buffer-total-bytes | client-id | producer-metrics.buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | DOUBLE_OBSERVABLE_GAUGE | -| 125 | producer-metrics | bufferpool-wait-ratio | client-id | producer-metrics.bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 126 | producer-metrics | bufferpool-wait-time-total | client-id | producer-metrics.bufferpool-wait-time-total | The total time an appender waits for space allocation. | DOUBLE_OBSERVABLE_COUNTER | -| 127 | producer-metrics | compression-rate-avg | client-id | producer-metrics.compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | -| 128 | producer-metrics | connection-close-rate | client-id | producer-metrics.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 129 | producer-metrics | connection-close-total | client-id | producer-metrics.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | -| 130 | producer-metrics | connection-count | client-id | producer-metrics.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | -| 131 | producer-metrics | connection-creation-rate | client-id | producer-metrics.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 132 | producer-metrics | connection-creation-total | client-id | producer-metrics.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | -| 133 | producer-metrics | failed-authentication-rate | client-id | producer-metrics.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 134 | producer-metrics | failed-authentication-total | client-id | producer-metrics.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | -| 135 | producer-metrics | failed-reauthentication-rate | client-id | producer-metrics.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 136 | producer-metrics | failed-reauthentication-total | client-id | producer-metrics.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 137 | producer-metrics | incoming-byte-rate | client-id | producer-metrics.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 138 | producer-metrics | incoming-byte-total | client-id | producer-metrics.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 139 | producer-metrics | io-ratio | client-id | producer-metrics.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 140 | producer-metrics | io-time-ns-avg | client-id | producer-metrics.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 141 | producer-metrics | io-wait-ratio | client-id | producer-metrics.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 142 | producer-metrics | io-wait-time-ns-avg | client-id | producer-metrics.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 143 | producer-metrics | io-waittime-total | client-id | producer-metrics.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | -| 144 | producer-metrics | iotime-total | client-id | producer-metrics.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | -| 145 | producer-metrics | metadata-age | client-id | producer-metrics.metadata-age | The age in seconds of the current producer metadata being used. | DOUBLE_OBSERVABLE_GAUGE | -| 146 | producer-metrics | network-io-rate | client-id | producer-metrics.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 147 | producer-metrics | network-io-total | client-id | producer-metrics.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | -| 148 | producer-metrics | outgoing-byte-rate | client-id | producer-metrics.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 149 | producer-metrics | outgoing-byte-total | client-id | producer-metrics.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 150 | producer-metrics | produce-throttle-time-avg | client-id | producer-metrics.produce-throttle-time-avg | The average time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | -| 151 | producer-metrics | produce-throttle-time-max | client-id | producer-metrics.produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | -| 152 | producer-metrics | reauthentication-latency-avg | client-id | producer-metrics.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 153 | producer-metrics | reauthentication-latency-max | client-id | producer-metrics.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 154 | producer-metrics | record-error-rate | client-id | producer-metrics.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 155 | producer-metrics | record-error-total | client-id | producer-metrics.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | -| 156 | producer-metrics | record-queue-time-avg | client-id | producer-metrics.record-queue-time-avg | The average time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | -| 157 | producer-metrics | record-queue-time-max | client-id | producer-metrics.record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | -| 158 | producer-metrics | record-retry-rate | client-id | producer-metrics.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 159 | producer-metrics | record-retry-total | client-id | producer-metrics.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | -| 160 | producer-metrics | record-send-rate | client-id | producer-metrics.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 161 | producer-metrics | record-send-total | client-id | producer-metrics.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | -| 162 | producer-metrics | record-size-avg | client-id | producer-metrics.record-size-avg | The average record size | DOUBLE_OBSERVABLE_GAUGE | -| 163 | producer-metrics | record-size-max | client-id | producer-metrics.record-size-max | The maximum record size | DOUBLE_OBSERVABLE_GAUGE | -| 164 | producer-metrics | records-per-request-avg | client-id | producer-metrics.records-per-request-avg | The average number of records per request. | DOUBLE_OBSERVABLE_GAUGE | -| 165 | producer-metrics | request-latency-avg | client-id | producer-metrics.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 166 | producer-metrics | request-latency-max | client-id | producer-metrics.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 167 | producer-metrics | request-rate | client-id | producer-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 168 | producer-metrics | request-size-avg | client-id | producer-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 169 | producer-metrics | request-size-max | client-id | producer-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 170 | producer-metrics | request-total | client-id | producer-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 171 | producer-metrics | requests-in-flight | client-id | producer-metrics.requests-in-flight | The current number of in-flight requests awaiting a response. | DOUBLE_OBSERVABLE_GAUGE | -| 172 | producer-metrics | response-rate | client-id | producer-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 173 | producer-metrics | response-total | client-id | producer-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 174 | producer-metrics | select-rate | client-id | producer-metrics.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 175 | producer-metrics | select-total | client-id | producer-metrics.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | -| 176 | producer-metrics | successful-authentication-no-reauth-total | client-id | producer-metrics.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | -| 177 | producer-metrics | successful-authentication-rate | client-id | producer-metrics.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 178 | producer-metrics | successful-authentication-total | client-id | producer-metrics.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | -| 179 | producer-metrics | successful-reauthentication-rate | client-id | producer-metrics.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 180 | producer-metrics | successful-reauthentication-total | client-id | producer-metrics.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 181 | producer-metrics | waiting-threads | client-id | producer-metrics.waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | DOUBLE_OBSERVABLE_GAUGE | -| 182 | producer-node-metrics | incoming-byte-rate | client-id,node-id | producer-node-metrics.incoming-byte-rate | The number of incoming bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 183 | producer-node-metrics | incoming-byte-total | client-id,node-id | producer-node-metrics.incoming-byte-total | The total number of incoming bytes | DOUBLE_OBSERVABLE_COUNTER | -| 184 | producer-node-metrics | outgoing-byte-rate | client-id,node-id | producer-node-metrics.outgoing-byte-rate | The number of outgoing bytes per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 185 | producer-node-metrics | outgoing-byte-total | client-id,node-id | producer-node-metrics.outgoing-byte-total | The total number of outgoing bytes | DOUBLE_OBSERVABLE_COUNTER | -| 186 | producer-node-metrics | request-latency-avg | client-id,node-id | producer-node-metrics.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | -| 187 | producer-node-metrics | request-latency-max | client-id,node-id | producer-node-metrics.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | -| 188 | producer-node-metrics | request-rate | client-id,node-id | producer-node-metrics.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 189 | producer-node-metrics | request-size-avg | client-id,node-id | producer-node-metrics.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 190 | producer-node-metrics | request-size-max | client-id,node-id | producer-node-metrics.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 191 | producer-node-metrics | request-total | client-id,node-id | producer-node-metrics.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 192 | producer-node-metrics | response-rate | client-id,node-id | producer-node-metrics.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 193 | producer-node-metrics | response-total | client-id,node-id | producer-node-metrics.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 194 | producer-topic-metrics | byte-rate | client-id,topic | producer-topic-metrics.byte-rate | The average number of bytes sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 195 | producer-topic-metrics | byte-total | client-id,topic | producer-topic-metrics.byte-total | The total number of bytes sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | -| 196 | producer-topic-metrics | compression-rate | client-id,topic | producer-topic-metrics.compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | -| 197 | producer-topic-metrics | record-error-rate | client-id,topic | producer-topic-metrics.record-error-rate | The average per-second number of record sends that resulted in errors for a topic | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 198 | producer-topic-metrics | record-error-total | client-id,topic | producer-topic-metrics.record-error-total | The total number of record sends that resulted in errors for a topic | DOUBLE_OBSERVABLE_COUNTER | -| 199 | producer-topic-metrics | record-retry-rate | client-id,topic | producer-topic-metrics.record-retry-rate | The average per-second number of retried record sends for a topic | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 200 | producer-topic-metrics | record-retry-total | client-id,topic | producer-topic-metrics.record-retry-total | The total number of retried record sends for a topic | DOUBLE_OBSERVABLE_COUNTER | -| 201 | producer-topic-metrics | record-send-rate | client-id,topic | producer-topic-metrics.record-send-rate | The average number of records sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 202 | producer-topic-metrics | record-send-total | client-id,topic | producer-topic-metrics.record-send-total | The total number of records sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | +| 50 | consumer-fetch-manager-metrics | records-consumed-rate | client-id | kafka.consumer.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 51 | consumer-fetch-manager-metrics | records-consumed-rate | client-id,topic | kafka.consumer.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 52 | consumer-fetch-manager-metrics | records-consumed-total | client-id,topic | kafka.consumer.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | +| 53 | consumer-fetch-manager-metrics | records-consumed-total | client-id | kafka.consumer.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | +| 54 | consumer-fetch-manager-metrics | records-lag | client-id,topic,partition | kafka.consumer.records-lag | The latest lag of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 55 | consumer-fetch-manager-metrics | records-lag-avg | client-id,topic,partition | kafka.consumer.records-lag-avg | The average lag of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 56 | consumer-fetch-manager-metrics | records-lag-max | client-id,topic,partition | kafka.consumer.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 57 | consumer-fetch-manager-metrics | records-lag-max | client-id | kafka.consumer.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 58 | consumer-fetch-manager-metrics | records-lead | client-id,topic,partition | kafka.consumer.records-lead | The latest lead of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 59 | consumer-fetch-manager-metrics | records-lead-avg | client-id,topic,partition | kafka.consumer.records-lead-avg | The average lead of the partition | DOUBLE_OBSERVABLE_GAUGE | +| 60 | consumer-fetch-manager-metrics | records-lead-min | client-id | kafka.consumer.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 61 | consumer-fetch-manager-metrics | records-lead-min | client-id,topic,partition | kafka.consumer.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | +| 62 | consumer-fetch-manager-metrics | records-per-request-avg | client-id,topic | kafka.consumer.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | +| 63 | consumer-fetch-manager-metrics | records-per-request-avg | client-id | kafka.consumer.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | +| 64 | consumer-metrics | connection-close-rate | client-id | kafka.consumer.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 65 | consumer-metrics | connection-close-total | client-id | kafka.consumer.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | +| 66 | consumer-metrics | connection-count | client-id | kafka.consumer.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | +| 67 | consumer-metrics | connection-creation-rate | client-id | kafka.consumer.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 68 | consumer-metrics | connection-creation-total | client-id | kafka.consumer.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | +| 69 | consumer-metrics | failed-authentication-rate | client-id | kafka.consumer.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 70 | consumer-metrics | failed-authentication-total | client-id | kafka.consumer.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | +| 71 | consumer-metrics | failed-reauthentication-rate | client-id | kafka.consumer.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 72 | consumer-metrics | failed-reauthentication-total | client-id | kafka.consumer.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 73 | consumer-metrics | incoming-byte-rate | client-id | kafka.consumer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 74 | consumer-metrics | incoming-byte-total | client-id | kafka.consumer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 75 | consumer-metrics | io-ratio | client-id | kafka.consumer.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 76 | consumer-metrics | io-time-ns-avg | client-id | kafka.consumer.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 77 | consumer-metrics | io-wait-ratio | client-id | kafka.consumer.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 78 | consumer-metrics | io-wait-time-ns-avg | client-id | kafka.consumer.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 79 | consumer-metrics | io-waittime-total | client-id | kafka.consumer.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | +| 80 | consumer-metrics | iotime-total | client-id | kafka.consumer.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | +| 81 | consumer-metrics | last-poll-seconds-ago | client-id | kafka.consumer.last-poll-seconds-ago | The number of seconds since the last poll() invocation. | DOUBLE_OBSERVABLE_GAUGE | +| 82 | consumer-metrics | network-io-rate | client-id | kafka.consumer.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 83 | consumer-metrics | network-io-total | client-id | kafka.consumer.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | +| 84 | consumer-metrics | outgoing-byte-rate | client-id | kafka.consumer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 85 | consumer-metrics | outgoing-byte-total | client-id | kafka.consumer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 86 | consumer-metrics | poll-idle-ratio-avg | client-id | kafka.consumer.poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | DOUBLE_OBSERVABLE_GAUGE | +| 87 | consumer-metrics | reauthentication-latency-avg | client-id | kafka.consumer.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 88 | consumer-metrics | reauthentication-latency-max | client-id | kafka.consumer.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 89 | consumer-metrics | request-rate | client-id | kafka.consumer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 90 | consumer-metrics | request-size-avg | client-id | kafka.consumer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 91 | consumer-metrics | request-size-max | client-id | kafka.consumer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 92 | consumer-metrics | request-total | client-id | kafka.consumer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 93 | consumer-metrics | response-rate | client-id | kafka.consumer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 94 | consumer-metrics | response-total | client-id | kafka.consumer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 95 | consumer-metrics | select-rate | client-id | kafka.consumer.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 96 | consumer-metrics | select-total | client-id | kafka.consumer.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | +| 97 | consumer-metrics | successful-authentication-no-reauth-total | client-id | kafka.consumer.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | +| 98 | consumer-metrics | successful-authentication-rate | client-id | kafka.consumer.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 99 | consumer-metrics | successful-authentication-total | client-id | kafka.consumer.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | +| 100 | consumer-metrics | successful-reauthentication-rate | client-id | kafka.consumer.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 101 | consumer-metrics | successful-reauthentication-total | client-id | kafka.consumer.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 102 | consumer-metrics | time-between-poll-avg | client-id | kafka.consumer.time-between-poll-avg | The average delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | +| 103 | consumer-metrics | time-between-poll-max | client-id | kafka.consumer.time-between-poll-max | The max delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | +| 104 | consumer-node-metrics | incoming-byte-rate | client-id,node-id | kafka.consumer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 105 | consumer-node-metrics | incoming-byte-total | client-id,node-id | kafka.consumer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 106 | consumer-node-metrics | outgoing-byte-rate | client-id,node-id | kafka.consumer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 107 | consumer-node-metrics | outgoing-byte-total | client-id,node-id | kafka.consumer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 108 | consumer-node-metrics | request-latency-avg | client-id,node-id | kafka.consumer.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | +| 109 | consumer-node-metrics | request-latency-max | client-id,node-id | kafka.consumer.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | +| 110 | consumer-node-metrics | request-rate | client-id,node-id | kafka.consumer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 111 | consumer-node-metrics | request-size-avg | client-id,node-id | kafka.consumer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 112 | consumer-node-metrics | request-size-max | client-id,node-id | kafka.consumer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 113 | consumer-node-metrics | request-total | client-id,node-id | kafka.consumer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 114 | consumer-node-metrics | response-rate | client-id,node-id | kafka.consumer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 115 | consumer-node-metrics | response-total | client-id,node-id | kafka.consumer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 116 | kafka-metrics-count | count | client-id | | | | +| 117 | producer-metrics | batch-size-avg | client-id | kafka.producer.batch-size-avg | The average number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | +| 118 | producer-metrics | batch-size-max | client-id | kafka.producer.batch-size-max | The max number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | +| 119 | producer-metrics | batch-split-rate | client-id | kafka.producer.batch-split-rate | The average number of batch splits per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 120 | producer-metrics | batch-split-total | client-id | kafka.producer.batch-split-total | The total number of batch splits | DOUBLE_OBSERVABLE_COUNTER | +| 121 | producer-metrics | buffer-available-bytes | client-id | kafka.producer.buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | DOUBLE_OBSERVABLE_GAUGE | +| 122 | producer-metrics | buffer-exhausted-rate | client-id | kafka.producer.buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 123 | producer-metrics | buffer-exhausted-total | client-id | kafka.producer.buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_COUNTER | +| 124 | producer-metrics | buffer-total-bytes | client-id | kafka.producer.buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | DOUBLE_OBSERVABLE_GAUGE | +| 125 | producer-metrics | bufferpool-wait-ratio | client-id | kafka.producer.bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 126 | producer-metrics | bufferpool-wait-time-total | client-id | kafka.producer.bufferpool-wait-time-total | The total time an appender waits for space allocation. | DOUBLE_OBSERVABLE_COUNTER | +| 127 | producer-metrics | compression-rate-avg | client-id | kafka.producer.compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | +| 128 | producer-metrics | connection-close-rate | client-id | kafka.producer.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 129 | producer-metrics | connection-close-total | client-id | kafka.producer.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | +| 130 | producer-metrics | connection-count | client-id | kafka.producer.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | +| 131 | producer-metrics | connection-creation-rate | client-id | kafka.producer.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 132 | producer-metrics | connection-creation-total | client-id | kafka.producer.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | +| 133 | producer-metrics | failed-authentication-rate | client-id | kafka.producer.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 134 | producer-metrics | failed-authentication-total | client-id | kafka.producer.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | +| 135 | producer-metrics | failed-reauthentication-rate | client-id | kafka.producer.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 136 | producer-metrics | failed-reauthentication-total | client-id | kafka.producer.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 137 | producer-metrics | incoming-byte-rate | client-id | kafka.producer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 138 | producer-metrics | incoming-byte-total | client-id | kafka.producer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 139 | producer-metrics | io-ratio | client-id | kafka.producer.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 140 | producer-metrics | io-time-ns-avg | client-id | kafka.producer.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 141 | producer-metrics | io-wait-ratio | client-id | kafka.producer.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 142 | producer-metrics | io-wait-time-ns-avg | client-id | kafka.producer.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | +| 143 | producer-metrics | io-waittime-total | client-id | kafka.producer.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | +| 144 | producer-metrics | iotime-total | client-id | kafka.producer.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | +| 145 | producer-metrics | metadata-age | client-id | kafka.producer.metadata-age | The age in seconds of the current producer metadata being used. | DOUBLE_OBSERVABLE_GAUGE | +| 146 | producer-metrics | network-io-rate | client-id | kafka.producer.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 147 | producer-metrics | network-io-total | client-id | kafka.producer.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | +| 148 | producer-metrics | outgoing-byte-rate | client-id | kafka.producer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 149 | producer-metrics | outgoing-byte-total | client-id | kafka.producer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 150 | producer-metrics | produce-throttle-time-avg | client-id | kafka.producer.produce-throttle-time-avg | The average time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | +| 151 | producer-metrics | produce-throttle-time-max | client-id | kafka.producer.produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | +| 152 | producer-metrics | reauthentication-latency-avg | client-id | kafka.producer.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 153 | producer-metrics | reauthentication-latency-max | client-id | kafka.producer.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | +| 154 | producer-metrics | record-error-rate | client-id | kafka.producer.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 155 | producer-metrics | record-error-total | client-id | kafka.producer.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | +| 156 | producer-metrics | record-queue-time-avg | client-id | kafka.producer.record-queue-time-avg | The average time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | +| 157 | producer-metrics | record-queue-time-max | client-id | kafka.producer.record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | +| 158 | producer-metrics | record-retry-rate | client-id | kafka.producer.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 159 | producer-metrics | record-retry-total | client-id | kafka.producer.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | +| 160 | producer-metrics | record-send-rate | client-id | kafka.producer.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 161 | producer-metrics | record-send-total | client-id | kafka.producer.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | +| 162 | producer-metrics | record-size-avg | client-id | kafka.producer.record-size-avg | The average record size | DOUBLE_OBSERVABLE_GAUGE | +| 163 | producer-metrics | record-size-max | client-id | kafka.producer.record-size-max | The maximum record size | DOUBLE_OBSERVABLE_GAUGE | +| 164 | producer-metrics | records-per-request-avg | client-id | kafka.producer.records-per-request-avg | The average number of records per request. | DOUBLE_OBSERVABLE_GAUGE | +| 165 | producer-metrics | request-latency-avg | client-id | kafka.producer.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 166 | producer-metrics | request-latency-max | client-id | kafka.producer.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 167 | producer-metrics | request-rate | client-id | kafka.producer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 168 | producer-metrics | request-size-avg | client-id | kafka.producer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 169 | producer-metrics | request-size-max | client-id | kafka.producer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 170 | producer-metrics | request-total | client-id | kafka.producer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 171 | producer-metrics | requests-in-flight | client-id | kafka.producer.requests-in-flight | The current number of in-flight requests awaiting a response. | DOUBLE_OBSERVABLE_GAUGE | +| 172 | producer-metrics | response-rate | client-id | kafka.producer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 173 | producer-metrics | response-total | client-id | kafka.producer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 174 | producer-metrics | select-rate | client-id | kafka.producer.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 175 | producer-metrics | select-total | client-id | kafka.producer.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | +| 176 | producer-metrics | successful-authentication-no-reauth-total | client-id | kafka.producer.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | +| 177 | producer-metrics | successful-authentication-rate | client-id | kafka.producer.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 178 | producer-metrics | successful-authentication-total | client-id | kafka.producer.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | +| 179 | producer-metrics | successful-reauthentication-rate | client-id | kafka.producer.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 180 | producer-metrics | successful-reauthentication-total | client-id | kafka.producer.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | +| 181 | producer-metrics | waiting-threads | client-id | kafka.producer.waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | DOUBLE_OBSERVABLE_GAUGE | +| 182 | producer-node-metrics | incoming-byte-rate | client-id,node-id | kafka.producer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 183 | producer-node-metrics | incoming-byte-total | client-id,node-id | kafka.producer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | +| 184 | producer-node-metrics | outgoing-byte-rate | client-id,node-id | kafka.producer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 185 | producer-node-metrics | outgoing-byte-total | client-id,node-id | kafka.producer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | +| 186 | producer-node-metrics | request-latency-avg | client-id,node-id | kafka.producer.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 187 | producer-node-metrics | request-latency-max | client-id,node-id | kafka.producer.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | +| 188 | producer-node-metrics | request-rate | client-id,node-id | kafka.producer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 189 | producer-node-metrics | request-size-avg | client-id,node-id | kafka.producer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | +| 190 | producer-node-metrics | request-size-max | client-id,node-id | kafka.producer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | +| 191 | producer-node-metrics | request-total | client-id,node-id | kafka.producer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | +| 192 | producer-node-metrics | response-rate | client-id,node-id | kafka.producer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 193 | producer-node-metrics | response-total | client-id,node-id | kafka.producer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | +| 194 | producer-topic-metrics | byte-rate | client-id,topic | kafka.producer.byte-rate | The average number of bytes sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 195 | producer-topic-metrics | byte-total | client-id,topic | kafka.producer.byte-total | The total number of bytes sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | +| 196 | producer-topic-metrics | compression-rate | client-id,topic | kafka.producer.compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | +| 197 | producer-topic-metrics | record-error-rate | client-id,topic | kafka.producer.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 198 | producer-topic-metrics | record-error-total | client-id,topic | kafka.producer.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | +| 199 | producer-topic-metrics | record-retry-rate | client-id,topic | kafka.producer.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 200 | producer-topic-metrics | record-retry-total | client-id,topic | kafka.producer.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | +| 201 | producer-topic-metrics | record-send-rate | client-id,topic | kafka.producer.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | +| 202 | producer-topic-metrics | record-send-total | client-id,topic | kafka.producer.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index cda51cf800c7..c54818158edb 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -58,7 +58,8 @@ static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) if (kafkaMetricId.getMeasureable() == null) { return null; } - Optional matchingGroup = groups.stream().filter(group -> kafkaMetricId.getGroup().contains(group)).findFirst(); + Optional matchingGroup = + groups.stream().filter(group -> kafkaMetricId.getGroup().contains(group)).findFirst(); // Only map metrics that have a matching group if (!matchingGroup.isPresent()) { return null; From a67afad4cc1a43102add5b8db5b787e6fc5d0d01 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 21 Jun 2022 18:32:32 -0500 Subject: [PATCH 06/17] PR feedback --- instrumentation/kafka/kafka-clients/README.md | 229 +--------- .../testing/build.gradle.kts | 4 +- .../kafkaclients/KafkaClientBaseTest.groovy | 4 +- .../OpenTelemetryKafkaMetricsTest.java | 396 +++++++++--------- .../kafka-clients-common/library/README.md | 226 ++++++++++ .../kafkaclients/KafkaMetricRegistry.java | 10 +- .../OpenTelemetryKafkaMetrics.java | 60 +-- 7 files changed, 476 insertions(+), 453 deletions(-) create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md diff --git a/instrumentation/kafka/kafka-clients/README.md b/instrumentation/kafka/kafka-clients/README.md index 4929ac092e77..170efde43cf8 100644 --- a/instrumentation/kafka/kafka-clients/README.md +++ b/instrumentation/kafka/kafka-clients/README.md @@ -3,231 +3,4 @@ | System property | Type | Default | Description | |---|---|---|---| | `otel.instrumentation.kafka.experimental-span-attributes` | Boolean | `false` | Enable the capture of experimental span attributes. | -| `otel.instrumentation.kafka.client-propagation.enabled` | Boolean | `true` | Enables remote context propagation via Kafka message headers. | - -# Kafka Metrics - -The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface. -OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry. - -To use, configure `OpenTelemetryKafkaMetrics` with an OpenTelemetry instance -via `OpenTelemetryKafkaMetrics#setOpenTelemetry(OpenTelemetry)`, and include a reference to this -class in kafka producer or consumer configuration, i.e.: - -```java -Map config = new HashMap<>(); -config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); -config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); -... -try (KafkaProducer producer = new KafkaProducer<>(config)) { ... } -``` - -The following table shows the full set of metrics exposed by the kafka client, and the corresponding -OpenTelemetry metric each maps to (if available). Empty values in the Instrument Name, Instrument -Description, etc column indicates there is no registered mapping for the metric and data is NOT -collected. - -| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | -|---|--------------|-------------|----------------|-----------------|------------------------|-----------------| -| 1 | app-info | commit-id | client-id | | | | -| 2 | app-info | start-time-ms | client-id | | | | -| 3 | app-info | version | client-id | | | | -| 4 | consumer-coordinator-metrics | assigned-partitions | client-id | kafka.consumer.assigned-partitions | The number of partitions currently assigned to this consumer | DOUBLE_OBSERVABLE_GAUGE | -| 5 | consumer-coordinator-metrics | commit-latency-avg | client-id | kafka.consumer.commit-latency-avg | The average time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | -| 6 | consumer-coordinator-metrics | commit-latency-max | client-id | kafka.consumer.commit-latency-max | The max time taken for a commit request | DOUBLE_OBSERVABLE_GAUGE | -| 7 | consumer-coordinator-metrics | commit-rate | client-id | kafka.consumer.commit-rate | The number of commit calls per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 8 | consumer-coordinator-metrics | commit-total | client-id | kafka.consumer.commit-total | The total number of commit calls | DOUBLE_OBSERVABLE_COUNTER | -| 9 | consumer-coordinator-metrics | failed-rebalance-rate-per-hour | client-id | kafka.consumer.failed-rebalance-rate-per-hour | The number of failed rebalance events per hour | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 10 | consumer-coordinator-metrics | failed-rebalance-total | client-id | kafka.consumer.failed-rebalance-total | The total number of failed rebalance events | DOUBLE_OBSERVABLE_COUNTER | -| 11 | consumer-coordinator-metrics | heartbeat-rate | client-id | kafka.consumer.heartbeat-rate | The number of heartbeats per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 12 | consumer-coordinator-metrics | heartbeat-response-time-max | client-id | kafka.consumer.heartbeat-response-time-max | The max time taken to receive a response to a heartbeat request | DOUBLE_OBSERVABLE_GAUGE | -| 13 | consumer-coordinator-metrics | heartbeat-total | client-id | kafka.consumer.heartbeat-total | The total number of heartbeats | DOUBLE_OBSERVABLE_COUNTER | -| 14 | consumer-coordinator-metrics | join-rate | client-id | kafka.consumer.join-rate | The number of group joins per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 15 | consumer-coordinator-metrics | join-time-avg | client-id | kafka.consumer.join-time-avg | The average time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | -| 16 | consumer-coordinator-metrics | join-time-max | client-id | kafka.consumer.join-time-max | The max time taken for a group rejoin | DOUBLE_OBSERVABLE_GAUGE | -| 17 | consumer-coordinator-metrics | join-total | client-id | kafka.consumer.join-total | The total number of group joins | DOUBLE_OBSERVABLE_COUNTER | -| 18 | consumer-coordinator-metrics | last-heartbeat-seconds-ago | client-id | kafka.consumer.last-heartbeat-seconds-ago | The number of seconds since the last coordinator heartbeat was sent | DOUBLE_OBSERVABLE_GAUGE | -| 19 | consumer-coordinator-metrics | last-rebalance-seconds-ago | client-id | kafka.consumer.last-rebalance-seconds-ago | The number of seconds since the last successful rebalance event | DOUBLE_OBSERVABLE_GAUGE | -| 20 | consumer-coordinator-metrics | partition-assigned-latency-avg | client-id | kafka.consumer.partition-assigned-latency-avg | The average time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 21 | consumer-coordinator-metrics | partition-assigned-latency-max | client-id | kafka.consumer.partition-assigned-latency-max | The max time taken for a partition-assigned rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 22 | consumer-coordinator-metrics | partition-lost-latency-avg | client-id | kafka.consumer.partition-lost-latency-avg | The average time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 23 | consumer-coordinator-metrics | partition-lost-latency-max | client-id | kafka.consumer.partition-lost-latency-max | The max time taken for a partition-lost rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 24 | consumer-coordinator-metrics | partition-revoked-latency-avg | client-id | kafka.consumer.partition-revoked-latency-avg | The average time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 25 | consumer-coordinator-metrics | partition-revoked-latency-max | client-id | kafka.consumer.partition-revoked-latency-max | The max time taken for a partition-revoked rebalance listener callback | DOUBLE_OBSERVABLE_GAUGE | -| 26 | consumer-coordinator-metrics | rebalance-latency-avg | client-id | kafka.consumer.rebalance-latency-avg | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | -| 27 | consumer-coordinator-metrics | rebalance-latency-max | client-id | kafka.consumer.rebalance-latency-max | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_GAUGE | -| 28 | consumer-coordinator-metrics | rebalance-latency-total | client-id | kafka.consumer.rebalance-latency-total | The total number of milliseconds this consumer has spent in successful rebalances since creation | DOUBLE_OBSERVABLE_COUNTER | -| 29 | consumer-coordinator-metrics | rebalance-rate-per-hour | client-id | kafka.consumer.rebalance-rate-per-hour | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 30 | consumer-coordinator-metrics | rebalance-total | client-id | kafka.consumer.rebalance-total | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | DOUBLE_OBSERVABLE_COUNTER | -| 31 | consumer-coordinator-metrics | sync-rate | client-id | kafka.consumer.sync-rate | The number of group syncs per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 32 | consumer-coordinator-metrics | sync-time-avg | client-id | kafka.consumer.sync-time-avg | The average time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | -| 33 | consumer-coordinator-metrics | sync-time-max | client-id | kafka.consumer.sync-time-max | The max time taken for a group sync | DOUBLE_OBSERVABLE_GAUGE | -| 34 | consumer-coordinator-metrics | sync-total | client-id | kafka.consumer.sync-total | The total number of group syncs | DOUBLE_OBSERVABLE_COUNTER | -| 35 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id | kafka.consumer.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 36 | consumer-fetch-manager-metrics | bytes-consumed-rate | client-id,topic | kafka.consumer.bytes-consumed-rate | The average number of bytes consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 37 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id,topic | kafka.consumer.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | -| 38 | consumer-fetch-manager-metrics | bytes-consumed-total | client-id | kafka.consumer.bytes-consumed-total | The total number of bytes consumed | DOUBLE_OBSERVABLE_COUNTER | -| 39 | consumer-fetch-manager-metrics | fetch-latency-avg | client-id | kafka.consumer.fetch-latency-avg | The average time taken for a fetch request. | DOUBLE_OBSERVABLE_GAUGE | -| 40 | consumer-fetch-manager-metrics | fetch-latency-max | client-id | kafka.consumer.fetch-latency-max | The max time taken for any fetch request. | DOUBLE_OBSERVABLE_GAUGE | -| 41 | consumer-fetch-manager-metrics | fetch-rate | client-id | kafka.consumer.fetch-rate | The number of fetch requests per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 42 | consumer-fetch-manager-metrics | fetch-size-avg | client-id,topic | kafka.consumer.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 43 | consumer-fetch-manager-metrics | fetch-size-avg | client-id | kafka.consumer.fetch-size-avg | The average number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 44 | consumer-fetch-manager-metrics | fetch-size-max | client-id,topic | kafka.consumer.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 45 | consumer-fetch-manager-metrics | fetch-size-max | client-id | kafka.consumer.fetch-size-max | The maximum number of bytes fetched per request | DOUBLE_OBSERVABLE_GAUGE | -| 46 | consumer-fetch-manager-metrics | fetch-throttle-time-avg | client-id | kafka.consumer.fetch-throttle-time-avg | The average throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | -| 47 | consumer-fetch-manager-metrics | fetch-throttle-time-max | client-id | kafka.consumer.fetch-throttle-time-max | The maximum throttle time in ms | DOUBLE_OBSERVABLE_GAUGE | -| 48 | consumer-fetch-manager-metrics | fetch-total | client-id | kafka.consumer.fetch-total | The total number of fetch requests. | DOUBLE_OBSERVABLE_COUNTER | -| 49 | consumer-fetch-manager-metrics | preferred-read-replica | client-id,topic,partition | | | | -| 50 | consumer-fetch-manager-metrics | records-consumed-rate | client-id | kafka.consumer.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 51 | consumer-fetch-manager-metrics | records-consumed-rate | client-id,topic | kafka.consumer.records-consumed-rate | The average number of records consumed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 52 | consumer-fetch-manager-metrics | records-consumed-total | client-id,topic | kafka.consumer.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | -| 53 | consumer-fetch-manager-metrics | records-consumed-total | client-id | kafka.consumer.records-consumed-total | The total number of records consumed | DOUBLE_OBSERVABLE_COUNTER | -| 54 | consumer-fetch-manager-metrics | records-lag | client-id,topic,partition | kafka.consumer.records-lag | The latest lag of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 55 | consumer-fetch-manager-metrics | records-lag-avg | client-id,topic,partition | kafka.consumer.records-lag-avg | The average lag of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 56 | consumer-fetch-manager-metrics | records-lag-max | client-id,topic,partition | kafka.consumer.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 57 | consumer-fetch-manager-metrics | records-lag-max | client-id | kafka.consumer.records-lag-max | The maximum lag in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 58 | consumer-fetch-manager-metrics | records-lead | client-id,topic,partition | kafka.consumer.records-lead | The latest lead of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 59 | consumer-fetch-manager-metrics | records-lead-avg | client-id,topic,partition | kafka.consumer.records-lead-avg | The average lead of the partition | DOUBLE_OBSERVABLE_GAUGE | -| 60 | consumer-fetch-manager-metrics | records-lead-min | client-id | kafka.consumer.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 61 | consumer-fetch-manager-metrics | records-lead-min | client-id,topic,partition | kafka.consumer.records-lead-min | The minimum lead in terms of number of records for any partition in this window | DOUBLE_OBSERVABLE_GAUGE | -| 62 | consumer-fetch-manager-metrics | records-per-request-avg | client-id,topic | kafka.consumer.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | -| 63 | consumer-fetch-manager-metrics | records-per-request-avg | client-id | kafka.consumer.records-per-request-avg | The average number of records in each request | DOUBLE_OBSERVABLE_GAUGE | -| 64 | consumer-metrics | connection-close-rate | client-id | kafka.consumer.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 65 | consumer-metrics | connection-close-total | client-id | kafka.consumer.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | -| 66 | consumer-metrics | connection-count | client-id | kafka.consumer.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | -| 67 | consumer-metrics | connection-creation-rate | client-id | kafka.consumer.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 68 | consumer-metrics | connection-creation-total | client-id | kafka.consumer.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | -| 69 | consumer-metrics | failed-authentication-rate | client-id | kafka.consumer.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 70 | consumer-metrics | failed-authentication-total | client-id | kafka.consumer.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | -| 71 | consumer-metrics | failed-reauthentication-rate | client-id | kafka.consumer.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 72 | consumer-metrics | failed-reauthentication-total | client-id | kafka.consumer.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 73 | consumer-metrics | incoming-byte-rate | client-id | kafka.consumer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 74 | consumer-metrics | incoming-byte-total | client-id | kafka.consumer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 75 | consumer-metrics | io-ratio | client-id | kafka.consumer.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 76 | consumer-metrics | io-time-ns-avg | client-id | kafka.consumer.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 77 | consumer-metrics | io-wait-ratio | client-id | kafka.consumer.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 78 | consumer-metrics | io-wait-time-ns-avg | client-id | kafka.consumer.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 79 | consumer-metrics | io-waittime-total | client-id | kafka.consumer.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | -| 80 | consumer-metrics | iotime-total | client-id | kafka.consumer.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | -| 81 | consumer-metrics | last-poll-seconds-ago | client-id | kafka.consumer.last-poll-seconds-ago | The number of seconds since the last poll() invocation. | DOUBLE_OBSERVABLE_GAUGE | -| 82 | consumer-metrics | network-io-rate | client-id | kafka.consumer.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 83 | consumer-metrics | network-io-total | client-id | kafka.consumer.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | -| 84 | consumer-metrics | outgoing-byte-rate | client-id | kafka.consumer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 85 | consumer-metrics | outgoing-byte-total | client-id | kafka.consumer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 86 | consumer-metrics | poll-idle-ratio-avg | client-id | kafka.consumer.poll-idle-ratio-avg | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | DOUBLE_OBSERVABLE_GAUGE | -| 87 | consumer-metrics | reauthentication-latency-avg | client-id | kafka.consumer.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 88 | consumer-metrics | reauthentication-latency-max | client-id | kafka.consumer.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 89 | consumer-metrics | request-rate | client-id | kafka.consumer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 90 | consumer-metrics | request-size-avg | client-id | kafka.consumer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 91 | consumer-metrics | request-size-max | client-id | kafka.consumer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 92 | consumer-metrics | request-total | client-id | kafka.consumer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 93 | consumer-metrics | response-rate | client-id | kafka.consumer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 94 | consumer-metrics | response-total | client-id | kafka.consumer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 95 | consumer-metrics | select-rate | client-id | kafka.consumer.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 96 | consumer-metrics | select-total | client-id | kafka.consumer.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | -| 97 | consumer-metrics | successful-authentication-no-reauth-total | client-id | kafka.consumer.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | -| 98 | consumer-metrics | successful-authentication-rate | client-id | kafka.consumer.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 99 | consumer-metrics | successful-authentication-total | client-id | kafka.consumer.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | -| 100 | consumer-metrics | successful-reauthentication-rate | client-id | kafka.consumer.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 101 | consumer-metrics | successful-reauthentication-total | client-id | kafka.consumer.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 102 | consumer-metrics | time-between-poll-avg | client-id | kafka.consumer.time-between-poll-avg | The average delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | -| 103 | consumer-metrics | time-between-poll-max | client-id | kafka.consumer.time-between-poll-max | The max delay between invocations of poll(). | DOUBLE_OBSERVABLE_GAUGE | -| 104 | consumer-node-metrics | incoming-byte-rate | client-id,node-id | kafka.consumer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 105 | consumer-node-metrics | incoming-byte-total | client-id,node-id | kafka.consumer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 106 | consumer-node-metrics | outgoing-byte-rate | client-id,node-id | kafka.consumer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 107 | consumer-node-metrics | outgoing-byte-total | client-id,node-id | kafka.consumer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 108 | consumer-node-metrics | request-latency-avg | client-id,node-id | kafka.consumer.request-latency-avg | | DOUBLE_OBSERVABLE_GAUGE | -| 109 | consumer-node-metrics | request-latency-max | client-id,node-id | kafka.consumer.request-latency-max | | DOUBLE_OBSERVABLE_GAUGE | -| 110 | consumer-node-metrics | request-rate | client-id,node-id | kafka.consumer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 111 | consumer-node-metrics | request-size-avg | client-id,node-id | kafka.consumer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 112 | consumer-node-metrics | request-size-max | client-id,node-id | kafka.consumer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 113 | consumer-node-metrics | request-total | client-id,node-id | kafka.consumer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 114 | consumer-node-metrics | response-rate | client-id,node-id | kafka.consumer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 115 | consumer-node-metrics | response-total | client-id,node-id | kafka.consumer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 116 | kafka-metrics-count | count | client-id | | | | -| 117 | producer-metrics | batch-size-avg | client-id | kafka.producer.batch-size-avg | The average number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | -| 118 | producer-metrics | batch-size-max | client-id | kafka.producer.batch-size-max | The max number of bytes sent per partition per-request. | DOUBLE_OBSERVABLE_GAUGE | -| 119 | producer-metrics | batch-split-rate | client-id | kafka.producer.batch-split-rate | The average number of batch splits per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 120 | producer-metrics | batch-split-total | client-id | kafka.producer.batch-split-total | The total number of batch splits | DOUBLE_OBSERVABLE_COUNTER | -| 121 | producer-metrics | buffer-available-bytes | client-id | kafka.producer.buffer-available-bytes | The total amount of buffer memory that is not being used (either unallocated or in the free list). | DOUBLE_OBSERVABLE_GAUGE | -| 122 | producer-metrics | buffer-exhausted-rate | client-id | kafka.producer.buffer-exhausted-rate | The average per-second number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 123 | producer-metrics | buffer-exhausted-total | client-id | kafka.producer.buffer-exhausted-total | The total number of record sends that are dropped due to buffer exhaustion | DOUBLE_OBSERVABLE_COUNTER | -| 124 | producer-metrics | buffer-total-bytes | client-id | kafka.producer.buffer-total-bytes | The maximum amount of buffer memory the client can use (whether or not it is currently used). | DOUBLE_OBSERVABLE_GAUGE | -| 125 | producer-metrics | bufferpool-wait-ratio | client-id | kafka.producer.bufferpool-wait-ratio | The fraction of time an appender waits for space allocation. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 126 | producer-metrics | bufferpool-wait-time-total | client-id | kafka.producer.bufferpool-wait-time-total | The total time an appender waits for space allocation. | DOUBLE_OBSERVABLE_COUNTER | -| 127 | producer-metrics | compression-rate-avg | client-id | kafka.producer.compression-rate-avg | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | -| 128 | producer-metrics | connection-close-rate | client-id | kafka.producer.connection-close-rate | The number of connections closed per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 129 | producer-metrics | connection-close-total | client-id | kafka.producer.connection-close-total | The total number of connections closed | DOUBLE_OBSERVABLE_COUNTER | -| 130 | producer-metrics | connection-count | client-id | kafka.producer.connection-count | The current number of active connections. | DOUBLE_OBSERVABLE_GAUGE | -| 131 | producer-metrics | connection-creation-rate | client-id | kafka.producer.connection-creation-rate | The number of new connections established per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 132 | producer-metrics | connection-creation-total | client-id | kafka.producer.connection-creation-total | The total number of new connections established | DOUBLE_OBSERVABLE_COUNTER | -| 133 | producer-metrics | failed-authentication-rate | client-id | kafka.producer.failed-authentication-rate | The number of connections with failed authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 134 | producer-metrics | failed-authentication-total | client-id | kafka.producer.failed-authentication-total | The total number of connections with failed authentication | DOUBLE_OBSERVABLE_COUNTER | -| 135 | producer-metrics | failed-reauthentication-rate | client-id | kafka.producer.failed-reauthentication-rate | The number of failed re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 136 | producer-metrics | failed-reauthentication-total | client-id | kafka.producer.failed-reauthentication-total | The total number of failed re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 137 | producer-metrics | incoming-byte-rate | client-id | kafka.producer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 138 | producer-metrics | incoming-byte-total | client-id | kafka.producer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 139 | producer-metrics | io-ratio | client-id | kafka.producer.io-ratio | The fraction of time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 140 | producer-metrics | io-time-ns-avg | client-id | kafka.producer.io-time-ns-avg | The average length of time for I/O per select call in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 141 | producer-metrics | io-wait-ratio | client-id | kafka.producer.io-wait-ratio | The fraction of time the I/O thread spent waiting | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 142 | producer-metrics | io-wait-time-ns-avg | client-id | kafka.producer.io-wait-time-ns-avg | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | DOUBLE_OBSERVABLE_GAUGE | -| 143 | producer-metrics | io-waittime-total | client-id | kafka.producer.io-waittime-total | The total time the I/O thread spent waiting | DOUBLE_OBSERVABLE_COUNTER | -| 144 | producer-metrics | iotime-total | client-id | kafka.producer.iotime-total | The total time the I/O thread spent doing I/O | DOUBLE_OBSERVABLE_COUNTER | -| 145 | producer-metrics | metadata-age | client-id | kafka.producer.metadata-age | The age in seconds of the current producer metadata being used. | DOUBLE_OBSERVABLE_GAUGE | -| 146 | producer-metrics | network-io-rate | client-id | kafka.producer.network-io-rate | The number of network operations (reads or writes) on all connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 147 | producer-metrics | network-io-total | client-id | kafka.producer.network-io-total | The total number of network operations (reads or writes) on all connections | DOUBLE_OBSERVABLE_COUNTER | -| 148 | producer-metrics | outgoing-byte-rate | client-id | kafka.producer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 149 | producer-metrics | outgoing-byte-total | client-id | kafka.producer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 150 | producer-metrics | produce-throttle-time-avg | client-id | kafka.producer.produce-throttle-time-avg | The average time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | -| 151 | producer-metrics | produce-throttle-time-max | client-id | kafka.producer.produce-throttle-time-max | The maximum time in ms a request was throttled by a broker | DOUBLE_OBSERVABLE_GAUGE | -| 152 | producer-metrics | reauthentication-latency-avg | client-id | kafka.producer.reauthentication-latency-avg | The average latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 153 | producer-metrics | reauthentication-latency-max | client-id | kafka.producer.reauthentication-latency-max | The max latency observed due to re-authentication | DOUBLE_OBSERVABLE_GAUGE | -| 154 | producer-metrics | record-error-rate | client-id | kafka.producer.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 155 | producer-metrics | record-error-total | client-id | kafka.producer.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | -| 156 | producer-metrics | record-queue-time-avg | client-id | kafka.producer.record-queue-time-avg | The average time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | -| 157 | producer-metrics | record-queue-time-max | client-id | kafka.producer.record-queue-time-max | The maximum time in ms record batches spent in the send buffer. | DOUBLE_OBSERVABLE_GAUGE | -| 158 | producer-metrics | record-retry-rate | client-id | kafka.producer.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 159 | producer-metrics | record-retry-total | client-id | kafka.producer.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | -| 160 | producer-metrics | record-send-rate | client-id | kafka.producer.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 161 | producer-metrics | record-send-total | client-id | kafka.producer.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | -| 162 | producer-metrics | record-size-avg | client-id | kafka.producer.record-size-avg | The average record size | DOUBLE_OBSERVABLE_GAUGE | -| 163 | producer-metrics | record-size-max | client-id | kafka.producer.record-size-max | The maximum record size | DOUBLE_OBSERVABLE_GAUGE | -| 164 | producer-metrics | records-per-request-avg | client-id | kafka.producer.records-per-request-avg | The average number of records per request. | DOUBLE_OBSERVABLE_GAUGE | -| 165 | producer-metrics | request-latency-avg | client-id | kafka.producer.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 166 | producer-metrics | request-latency-max | client-id | kafka.producer.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 167 | producer-metrics | request-rate | client-id | kafka.producer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 168 | producer-metrics | request-size-avg | client-id | kafka.producer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 169 | producer-metrics | request-size-max | client-id | kafka.producer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 170 | producer-metrics | request-total | client-id | kafka.producer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 171 | producer-metrics | requests-in-flight | client-id | kafka.producer.requests-in-flight | The current number of in-flight requests awaiting a response. | DOUBLE_OBSERVABLE_GAUGE | -| 172 | producer-metrics | response-rate | client-id | kafka.producer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 173 | producer-metrics | response-total | client-id | kafka.producer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 174 | producer-metrics | select-rate | client-id | kafka.producer.select-rate | The number of times the I/O layer checked for new I/O to perform per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 175 | producer-metrics | select-total | client-id | kafka.producer.select-total | The total number of times the I/O layer checked for new I/O to perform | DOUBLE_OBSERVABLE_COUNTER | -| 176 | producer-metrics | successful-authentication-no-reauth-total | client-id | kafka.producer.successful-authentication-no-reauth-total | The total number of connections with successful authentication where the client does not support re-authentication | DOUBLE_OBSERVABLE_COUNTER | -| 177 | producer-metrics | successful-authentication-rate | client-id | kafka.producer.successful-authentication-rate | The number of connections with successful authentication per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 178 | producer-metrics | successful-authentication-total | client-id | kafka.producer.successful-authentication-total | The total number of connections with successful authentication | DOUBLE_OBSERVABLE_COUNTER | -| 179 | producer-metrics | successful-reauthentication-rate | client-id | kafka.producer.successful-reauthentication-rate | The number of successful re-authentication of connections per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 180 | producer-metrics | successful-reauthentication-total | client-id | kafka.producer.successful-reauthentication-total | The total number of successful re-authentication of connections | DOUBLE_OBSERVABLE_COUNTER | -| 181 | producer-metrics | waiting-threads | client-id | kafka.producer.waiting-threads | The number of user threads blocked waiting for buffer memory to enqueue their records | DOUBLE_OBSERVABLE_GAUGE | -| 182 | producer-node-metrics | incoming-byte-rate | client-id,node-id | kafka.producer.incoming-byte-rate | The number of bytes read off all sockets per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 183 | producer-node-metrics | incoming-byte-total | client-id,node-id | kafka.producer.incoming-byte-total | The total number of bytes read off all sockets | DOUBLE_OBSERVABLE_COUNTER | -| 184 | producer-node-metrics | outgoing-byte-rate | client-id,node-id | kafka.producer.outgoing-byte-rate | The number of outgoing bytes sent to all servers per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 185 | producer-node-metrics | outgoing-byte-total | client-id,node-id | kafka.producer.outgoing-byte-total | The total number of outgoing bytes sent to all servers | DOUBLE_OBSERVABLE_COUNTER | -| 186 | producer-node-metrics | request-latency-avg | client-id,node-id | kafka.producer.request-latency-avg | The average request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 187 | producer-node-metrics | request-latency-max | client-id,node-id | kafka.producer.request-latency-max | The maximum request latency in ms | DOUBLE_OBSERVABLE_GAUGE | -| 188 | producer-node-metrics | request-rate | client-id,node-id | kafka.producer.request-rate | The number of requests sent per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 189 | producer-node-metrics | request-size-avg | client-id,node-id | kafka.producer.request-size-avg | The average size of requests sent. | DOUBLE_OBSERVABLE_GAUGE | -| 190 | producer-node-metrics | request-size-max | client-id,node-id | kafka.producer.request-size-max | The maximum size of any request sent. | DOUBLE_OBSERVABLE_GAUGE | -| 191 | producer-node-metrics | request-total | client-id,node-id | kafka.producer.request-total | The total number of requests sent | DOUBLE_OBSERVABLE_COUNTER | -| 192 | producer-node-metrics | response-rate | client-id,node-id | kafka.producer.response-rate | The number of responses received per second | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 193 | producer-node-metrics | response-total | client-id,node-id | kafka.producer.response-total | The total number of responses received | DOUBLE_OBSERVABLE_COUNTER | -| 194 | producer-topic-metrics | byte-rate | client-id,topic | kafka.producer.byte-rate | The average number of bytes sent per second for a topic. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 195 | producer-topic-metrics | byte-total | client-id,topic | kafka.producer.byte-total | The total number of bytes sent for a topic. | DOUBLE_OBSERVABLE_COUNTER | -| 196 | producer-topic-metrics | compression-rate | client-id,topic | kafka.producer.compression-rate | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | DOUBLE_OBSERVABLE_GAUGE | -| 197 | producer-topic-metrics | record-error-rate | client-id,topic | kafka.producer.record-error-rate | The average per-second number of record sends that resulted in errors | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 198 | producer-topic-metrics | record-error-total | client-id,topic | kafka.producer.record-error-total | The total number of record sends that resulted in errors | DOUBLE_OBSERVABLE_COUNTER | -| 199 | producer-topic-metrics | record-retry-rate | client-id,topic | kafka.producer.record-retry-rate | The average per-second number of retried record sends | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 200 | producer-topic-metrics | record-retry-total | client-id,topic | kafka.producer.record-retry-total | The total number of retried record sends | DOUBLE_OBSERVABLE_COUNTER | -| 201 | producer-topic-metrics | record-send-rate | client-id,topic | kafka.producer.record-send-rate | The average number of records sent per second. | DOUBLE_OBSERVABLE_UP_DOWN_COUNTER | -| 202 | producer-topic-metrics | record-send-total | client-id,topic | kafka.producer.record-send-total | The total number of records sent. | DOUBLE_OBSERVABLE_COUNTER | +| `otel.instrumentation.kafka.client-propagation.enabled` | Boolean | `true` | Enables remote context propagation via Kafka message headers. | \ No newline at end of file diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts index cab63599bcba..d48496577f4b 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts @@ -10,9 +10,7 @@ dependencies { implementation(project(":instrumentation:kafka:kafka-clients:kafka-clients-common:library")) implementation("org.testcontainers:kafka") + implementation("org.testcontainers:junit-jupiter") runtimeOnly("org.apache.kafka:kafka_2.13:2.8.1") - implementation("com.salesforce.kafka.test:kafka-junit5:3.2.3") { - exclude(module = "zookeeper") // conflicts with zookeeper from kafka_2.x - } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/groovy/io/opentelemetry/instrumentation/kafkaclients/KafkaClientBaseTest.groovy b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/groovy/io/opentelemetry/instrumentation/kafkaclients/KafkaClientBaseTest.groovy index 0d77d5daaaa3..052396fdd489 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/groovy/io/opentelemetry/instrumentation/kafkaclients/KafkaClientBaseTest.groovy +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/groovy/io/opentelemetry/instrumentation/kafkaclients/KafkaClientBaseTest.groovy @@ -6,6 +6,8 @@ package io.opentelemetry.instrumentation.kafkaclients import io.opentelemetry.instrumentation.test.InstrumentationSpecification +import org.testcontainers.utility.DockerImageName + import java.time.Duration import org.apache.kafka.clients.admin.AdminClient import org.apache.kafka.clients.admin.NewTopic @@ -46,7 +48,7 @@ abstract class KafkaClientBaseTest extends InstrumentationSpecification { static TopicPartition topicPartition = new TopicPartition(SHARED_TOPIC, 0) def setupSpec() { - kafka = new KafkaContainer() + kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3")) .withLogConsumer(new Slf4jLogConsumer(logger)) .waitingFor(Wait.forLogMessage(".*started \\(kafka.server.KafkaServer\\).*", 1)) .withStartupTimeout(Duration.ofMinutes(1)) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index d121f5161e9a..b7dcf128efa8 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -8,11 +8,11 @@ import static java.util.stream.Collectors.toSet; import static org.assertj.core.api.Assertions.assertThat; -import com.salesforce.kafka.test.junit5.SharedKafkaTestResource; import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension; import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension; import io.opentelemetry.sdk.metrics.data.MetricData; import java.nio.charset.StandardCharsets; +import java.time.Duration; import java.time.Instant; import java.util.Arrays; import java.util.HashMap; @@ -30,13 +30,27 @@ import org.apache.kafka.common.serialization.ByteArraySerializer; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.RegisterExtension; +import org.slf4j.LoggerFactory; +import org.testcontainers.containers.KafkaContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +@Testcontainers abstract class OpenTelemetryKafkaMetricsTest { private static final List TOPICS = Arrays.asList("foo", "bar", "baz", "qux"); private static final Random RANDOM = new Random(); - @RegisterExtension static final SharedKafkaTestResource kafka = new SharedKafkaTestResource(); + @Container + KafkaContainer kafka = + new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3")) + .withLogConsumer( + new Slf4jLogConsumer(LoggerFactory.getLogger(OpenTelemetryKafkaMetricsTest.class))) + .waitingFor(Wait.forLogMessage(".*started \\(kafka.server.KafkaServer\\).*", 1)) + .withStartupTimeout(Duration.ofMinutes(1)); @RegisterExtension static final InstrumentationExtension testing = LibraryInstrumentationExtension.create(); @@ -51,193 +65,193 @@ void observeMetrics() { Set expectedMetricNames = new HashSet<>( Arrays.asList( - "kafka.consumer.commit-latency-avg", - "kafka.consumer.commit-latency-max", - "kafka.consumer.commit-rate", - "kafka.consumer.commit-total", - "kafka.consumer.failed-rebalance-rate-per-hour", - "kafka.consumer.failed-rebalance-total", - "kafka.consumer.heartbeat-rate", - "kafka.consumer.heartbeat-response-time-max", - "kafka.consumer.heartbeat-total", - "kafka.consumer.join-rate", - "kafka.consumer.join-time-avg", - "kafka.consumer.join-time-max", - "kafka.consumer.join-total", - "kafka.consumer.last-heartbeat-seconds-ago", - "kafka.consumer.last-rebalance-seconds-ago", - "kafka.consumer.partition-assigned-latency-avg", - "kafka.consumer.partition-assigned-latency-max", - "kafka.consumer.partition-lost-latency-avg", - "kafka.consumer.partition-lost-latency-max", - "kafka.consumer.partition-revoked-latency-avg", - "kafka.consumer.partition-revoked-latency-max", - "kafka.consumer.rebalance-latency-avg", - "kafka.consumer.rebalance-latency-max", - "kafka.consumer.rebalance-latency-total", - "kafka.consumer.rebalance-rate-per-hour", - "kafka.consumer.rebalance-total", - "kafka.consumer.sync-rate", - "kafka.consumer.sync-time-avg", - "kafka.consumer.sync-time-max", - "kafka.consumer.sync-total", - "kafka.consumer.bytes-consumed-rate", - "kafka.consumer.bytes-consumed-total", - "kafka.consumer.fetch-latency-avg", - "kafka.consumer.fetch-latency-max", - "kafka.consumer.fetch-rate", - "kafka.consumer.fetch-size-avg", - "kafka.consumer.fetch-size-max", - "kafka.consumer.fetch-throttle-time-avg", - "kafka.consumer.fetch-throttle-time-max", - "kafka.consumer.fetch-total", - "kafka.consumer.records-consumed-rate", - "kafka.consumer.records-consumed-total", - "kafka.consumer.records-lag", - "kafka.consumer.records-lag-avg", - "kafka.consumer.records-lag-max", - "kafka.consumer.records-lead", - "kafka.consumer.records-lead-avg", - "kafka.consumer.records-lead-min", - "kafka.consumer.records-per-request-avg", - "kafka.consumer.connection-close-rate", - "kafka.consumer.connection-close-total", - "kafka.consumer.connection-count", - "kafka.consumer.connection-creation-rate", - "kafka.consumer.connection-creation-total", - "kafka.consumer.failed-authentication-rate", - "kafka.consumer.failed-authentication-total", - "kafka.consumer.failed-reauthentication-rate", - "kafka.consumer.failed-reauthentication-total", - "kafka.consumer.incoming-byte-rate", - "kafka.consumer.incoming-byte-total", - "kafka.consumer.io-ratio", - "kafka.consumer.io-time-ns-avg", - "kafka.consumer.io-wait-ratio", - "kafka.consumer.io-wait-time-ns-avg", - "kafka.consumer.io-waittime-total", - "kafka.consumer.iotime-total", - "kafka.consumer.last-poll-seconds-ago", - "kafka.consumer.network-io-rate", - "kafka.consumer.network-io-total", - "kafka.consumer.outgoing-byte-rate", - "kafka.consumer.outgoing-byte-total", - "kafka.consumer.poll-idle-ratio-avg", - "kafka.consumer.reauthentication-latency-avg", - "kafka.consumer.reauthentication-latency-max", - "kafka.consumer.request-rate", - "kafka.consumer.request-size-avg", - "kafka.consumer.request-size-max", - "kafka.consumer.request-total", - "kafka.consumer.response-rate", - "kafka.consumer.response-total", - "kafka.consumer.select-rate", - "kafka.consumer.select-total", - "kafka.consumer.successful-authentication-no-reauth-total", - "kafka.consumer.successful-authentication-rate", - "kafka.consumer.successful-authentication-total", - "kafka.consumer.successful-reauthentication-rate", - "kafka.consumer.successful-reauthentication-total", - "kafka.consumer.time-between-poll-avg", - "kafka.consumer.time-between-poll-max", - "kafka.consumer.incoming-byte-rate", - "kafka.consumer.incoming-byte-total", - "kafka.consumer.outgoing-byte-rate", - "kafka.consumer.outgoing-byte-total", - "kafka.consumer.request-latency-avg", - "kafka.consumer.request-latency-max", - "kafka.consumer.request-rate", - "kafka.consumer.request-size-avg", - "kafka.consumer.request-size-max", - "kafka.consumer.request-total", - "kafka.consumer.response-rate", - "kafka.consumer.response-total", - "kafka.producer.batch-size-avg", - "kafka.producer.batch-size-max", - "kafka.producer.batch-split-rate", - "kafka.producer.batch-split-total", - "kafka.producer.buffer-available-bytes", - "kafka.producer.buffer-exhausted-rate", - "kafka.producer.buffer-exhausted-total", - "kafka.producer.buffer-total-bytes", - "kafka.producer.bufferpool-wait-ratio", - "kafka.producer.bufferpool-wait-time-total", - "kafka.producer.compression-rate-avg", - "kafka.producer.connection-close-rate", - "kafka.producer.connection-close-total", - "kafka.producer.connection-count", - "kafka.producer.connection-creation-rate", - "kafka.producer.connection-creation-total", - "kafka.producer.failed-authentication-rate", - "kafka.producer.failed-authentication-total", - "kafka.producer.failed-reauthentication-rate", - "kafka.producer.failed-reauthentication-total", - "kafka.producer.incoming-byte-rate", - "kafka.producer.incoming-byte-total", - "kafka.producer.io-ratio", - "kafka.producer.io-time-ns-avg", - "kafka.producer.io-wait-ratio", - "kafka.producer.io-wait-time-ns-avg", - "kafka.producer.io-waittime-total", - "kafka.producer.iotime-total", - "kafka.producer.metadata-age", - "kafka.producer.network-io-rate", - "kafka.producer.network-io-total", - "kafka.producer.outgoing-byte-rate", - "kafka.producer.outgoing-byte-total", - "kafka.producer.produce-throttle-time-avg", - "kafka.producer.produce-throttle-time-max", - "kafka.producer.reauthentication-latency-avg", - "kafka.producer.reauthentication-latency-max", - "kafka.producer.record-error-rate", - "kafka.producer.record-error-total", - "kafka.producer.record-queue-time-avg", - "kafka.producer.record-queue-time-max", - "kafka.producer.record-retry-rate", - "kafka.producer.record-retry-total", - "kafka.producer.record-send-rate", - "kafka.producer.record-send-total", - "kafka.producer.record-size-avg", - "kafka.producer.record-size-max", - "kafka.producer.records-per-request-avg", - "kafka.producer.request-latency-avg", - "kafka.producer.request-latency-max", - "kafka.producer.request-rate", - "kafka.producer.request-size-avg", - "kafka.producer.request-size-max", - "kafka.producer.request-total", - "kafka.producer.requests-in-flight", - "kafka.producer.response-rate", - "kafka.producer.response-total", - "kafka.producer.select-rate", - "kafka.producer.select-total", - "kafka.producer.successful-authentication-no-reauth-total", - "kafka.producer.successful-authentication-rate", - "kafka.producer.successful-authentication-total", - "kafka.producer.successful-reauthentication-rate", - "kafka.producer.successful-reauthentication-total", - "kafka.producer.waiting-threads", - "kafka.producer.incoming-byte-rate", - "kafka.producer.incoming-byte-total", - "kafka.producer.outgoing-byte-rate", - "kafka.producer.outgoing-byte-total", - "kafka.producer.request-latency-avg", - "kafka.producer.request-latency-max", - "kafka.producer.request-rate", - "kafka.producer.request-size-avg", - "kafka.producer.request-size-max", - "kafka.producer.request-total", - "kafka.producer.response-rate", - "kafka.producer.response-total", - "kafka.producer.byte-rate", - "kafka.producer.byte-total", - "kafka.producer.compression-rate", - "kafka.producer.record-error-rate", - "kafka.producer.record-error-total", - "kafka.producer.record-retry-rate", - "kafka.producer.record-retry-total", - "kafka.producer.record-send-rate", - "kafka.producer.record-send-total")); + "kafka.consumer.commit_latency_avg", + "kafka.consumer.commit_latency_max", + "kafka.consumer.commit_rate", + "kafka.consumer.commit_total", + "kafka.consumer.failed_rebalance_rate_per_hour", + "kafka.consumer.failed_rebalance_total", + "kafka.consumer.heartbeat_rate", + "kafka.consumer.heartbeat_response_time_max", + "kafka.consumer.heartbeat_total", + "kafka.consumer.join_rate", + "kafka.consumer.join_time_avg", + "kafka.consumer.join_time_max", + "kafka.consumer.join_total", + "kafka.consumer.last_heartbeat_seconds_ago", + "kafka.consumer.last_rebalance_seconds_ago", + "kafka.consumer.partition_assigned_latency_avg", + "kafka.consumer.partition_assigned_latency_max", + "kafka.consumer.partition_lost_latency_avg", + "kafka.consumer.partition_lost_latency_max", + "kafka.consumer.partition_revoked_latency_avg", + "kafka.consumer.partition_revoked_latency_max", + "kafka.consumer.rebalance_latency_avg", + "kafka.consumer.rebalance_latency_max", + "kafka.consumer.rebalance_latency_total", + "kafka.consumer.rebalance_rate_per_hour", + "kafka.consumer.rebalance_total", + "kafka.consumer.sync_rate", + "kafka.consumer.sync_time_avg", + "kafka.consumer.sync_time_max", + "kafka.consumer.sync_total", + "kafka.consumer.bytes_consumed_rate", + "kafka.consumer.bytes_consumed_total", + "kafka.consumer.fetch_latency_avg", + "kafka.consumer.fetch_latency_max", + "kafka.consumer.fetch_rate", + "kafka.consumer.fetch_size_avg", + "kafka.consumer.fetch_size_max", + "kafka.consumer.fetch_throttle_time_avg", + "kafka.consumer.fetch_throttle_time_max", + "kafka.consumer.fetch_total", + "kafka.consumer.records_consumed_rate", + "kafka.consumer.records_consumed_total", + "kafka.consumer.records_lag", + "kafka.consumer.records_lag_avg", + "kafka.consumer.records_lag_max", + "kafka.consumer.records_lead", + "kafka.consumer.records_lead_avg", + "kafka.consumer.records_lead_min", + "kafka.consumer.records_per_request_avg", + "kafka.consumer.connection_close_rate", + "kafka.consumer.connection_close_total", + "kafka.consumer.connection_count", + "kafka.consumer.connection_creation_rate", + "kafka.consumer.connection_creation_total", + "kafka.consumer.failed_authentication_rate", + "kafka.consumer.failed_authentication_total", + "kafka.consumer.failed_reauthentication_rate", + "kafka.consumer.failed_reauthentication_total", + "kafka.consumer.incoming_byte_rate", + "kafka.consumer.incoming_byte_total", + "kafka.consumer.io_ratio", + "kafka.consumer.io_time_ns_avg", + "kafka.consumer.io_wait_ratio", + "kafka.consumer.io_wait_time_ns_avg", + "kafka.consumer.io_waittime_total", + "kafka.consumer.iotime_total", + "kafka.consumer.last_poll_seconds_ago", + "kafka.consumer.network_io_rate", + "kafka.consumer.network_io_total", + "kafka.consumer.outgoing_byte_rate", + "kafka.consumer.outgoing_byte_total", + "kafka.consumer.poll_idle_ratio_avg", + "kafka.consumer.reauthentication_latency_avg", + "kafka.consumer.reauthentication_latency_max", + "kafka.consumer.request_rate", + "kafka.consumer.request_size_avg", + "kafka.consumer.request_size_max", + "kafka.consumer.request_total", + "kafka.consumer.response_rate", + "kafka.consumer.response_total", + "kafka.consumer.select_rate", + "kafka.consumer.select_total", + "kafka.consumer.successful_authentication_no_reauth_total", + "kafka.consumer.successful_authentication_rate", + "kafka.consumer.successful_authentication_total", + "kafka.consumer.successful_reauthentication_rate", + "kafka.consumer.successful_reauthentication_total", + "kafka.consumer.time_between_poll_avg", + "kafka.consumer.time_between_poll_max", + "kafka.consumer.incoming_byte_rate", + "kafka.consumer.incoming_byte_total", + "kafka.consumer.outgoing_byte_rate", + "kafka.consumer.outgoing_byte_total", + "kafka.consumer.request_latency_avg", + "kafka.consumer.request_latency_max", + "kafka.consumer.request_rate", + "kafka.consumer.request_size_avg", + "kafka.consumer.request_size_max", + "kafka.consumer.request_total", + "kafka.consumer.response_rate", + "kafka.consumer.response_total", + "kafka.producer.batch_size_avg", + "kafka.producer.batch_size_max", + "kafka.producer.batch_split_rate", + "kafka.producer.batch_split_total", + "kafka.producer.buffer_available_bytes", + "kafka.producer.buffer_exhausted_rate", + "kafka.producer.buffer_exhausted_total", + "kafka.producer.buffer_total_bytes", + "kafka.producer.bufferpool_wait_ratio", + "kafka.producer.bufferpool_wait_time_total", + "kafka.producer.compression_rate_avg", + "kafka.producer.connection_close_rate", + "kafka.producer.connection_close_total", + "kafka.producer.connection_count", + "kafka.producer.connection_creation_rate", + "kafka.producer.connection_creation_total", + "kafka.producer.failed_authentication_rate", + "kafka.producer.failed_authentication_total", + "kafka.producer.failed_reauthentication_rate", + "kafka.producer.failed_reauthentication_total", + "kafka.producer.incoming_byte_rate", + "kafka.producer.incoming_byte_total", + "kafka.producer.io_ratio", + "kafka.producer.io_time_ns_avg", + "kafka.producer.io_wait_ratio", + "kafka.producer.io_wait_time_ns_avg", + "kafka.producer.io_waittime_total", + "kafka.producer.iotime_total", + "kafka.producer.metadata_age", + "kafka.producer.network_io_rate", + "kafka.producer.network_io_total", + "kafka.producer.outgoing_byte_rate", + "kafka.producer.outgoing_byte_total", + "kafka.producer.produce_throttle_time_avg", + "kafka.producer.produce_throttle_time_max", + "kafka.producer.reauthentication_latency_avg", + "kafka.producer.reauthentication_latency_max", + "kafka.producer.record_error_rate", + "kafka.producer.record_error_total", + "kafka.producer.record_queue_time_avg", + "kafka.producer.record_queue_time_max", + "kafka.producer.record_retry_rate", + "kafka.producer.record_retry_total", + "kafka.producer.record_send_rate", + "kafka.producer.record_send_total", + "kafka.producer.record_size_avg", + "kafka.producer.record_size_max", + "kafka.producer.records_per_request_avg", + "kafka.producer.request_latency_avg", + "kafka.producer.request_latency_max", + "kafka.producer.request_rate", + "kafka.producer.request_size_avg", + "kafka.producer.request_size_max", + "kafka.producer.request_total", + "kafka.producer.requests_in_flight", + "kafka.producer.response_rate", + "kafka.producer.response_total", + "kafka.producer.select_rate", + "kafka.producer.select_total", + "kafka.producer.successful_authentication_no_reauth_total", + "kafka.producer.successful_authentication_rate", + "kafka.producer.successful_authentication_total", + "kafka.producer.successful_reauthentication_rate", + "kafka.producer.successful_reauthentication_total", + "kafka.producer.waiting_threads", + "kafka.producer.incoming_byte_rate", + "kafka.producer.incoming_byte_total", + "kafka.producer.outgoing_byte_rate", + "kafka.producer.outgoing_byte_total", + "kafka.producer.request_latency_avg", + "kafka.producer.request_latency_max", + "kafka.producer.request_rate", + "kafka.producer.request_size_avg", + "kafka.producer.request_size_max", + "kafka.producer.request_total", + "kafka.producer.response_rate", + "kafka.producer.response_total", + "kafka.producer.byte_rate", + "kafka.producer.byte_total", + "kafka.producer.compression_rate", + "kafka.producer.record_error_rate", + "kafka.producer.record_error_total", + "kafka.producer.record_retry_rate", + "kafka.producer.record_retry_total", + "kafka.producer.record_send_rate", + "kafka.producer.record_send_total")); Set metricNames = testing.metrics().stream().map(MetricData::getName).collect(toSet()); @@ -252,7 +266,7 @@ void produceRecords() { // Register OpenTelemetryKafkaMetrics as reporter config.put( ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); - config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); + config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); config.put(ProducerConfig.CLIENT_ID_CONFIG, "sample-client-id"); config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); @@ -276,7 +290,7 @@ void consumeRecords() { // Register OpenTelemetryKafkaMetrics as reporter config.put( ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); - config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); + config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); config.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-group"); config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); config.put( diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md new file mode 100644 index 000000000000..832f550f217d --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md @@ -0,0 +1,226 @@ +# Kafka Metrics + +The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface. +OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry. + +To use, configure `OpenTelemetryKafkaMetrics` with an OpenTelemetry instance +via `OpenTelemetryKafkaMetrics#setOpenTelemetry(OpenTelemetry)`, and include a reference to this +class in kafka producer or consumer configuration, i.e.: + +```java +Map config = new HashMap<>(); +config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); +config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString()); +... +try (KafkaProducer producer = new KafkaProducer<>(config)) { ... } +``` + +The following table shows the full set of metrics exposed by the kafka client, and the corresponding +OpenTelemetry metric each maps to (if available). Empty values in the Instrument Name, Instrument +Description, etc column indicates there is no registered mapping for the metric and data is NOT +collected. + +| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | +|---|--------------|-------------|----------------|-----------------|------------------------|-----------------| +| 1 | `app-info` | `commit-id` | client-id | | | | +| 2 | `app-info` | `start-time-ms` | client-id | | | | +| 3 | `app-info` | `version` | client-id | | | | +| 4 | `consumer-coordinator-metrics` | `assigned-partitions` | client-id | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | +| 5 | `consumer-coordinator-metrics` | `commit-latency-avg` | client-id | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| 6 | `consumer-coordinator-metrics` | `commit-latency-max` | client-id | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| 7 | `consumer-coordinator-metrics` | `commit-rate` | client-id | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 8 | `consumer-coordinator-metrics` | `commit-total` | client-id | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | +| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | client-id | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 10 | `consumer-coordinator-metrics` | `failed-rebalance-total` | client-id | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | +| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | client-id | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 12 | `consumer-coordinator-metrics` | `heartbeat-response-time-max` | client-id | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | +| 13 | `consumer-coordinator-metrics` | `heartbeat-total` | client-id | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | +| 14 | `consumer-coordinator-metrics` | `join-rate` | client-id | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 15 | `consumer-coordinator-metrics` | `join-time-avg` | client-id | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| 16 | `consumer-coordinator-metrics` | `join-time-max` | client-id | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| 17 | `consumer-coordinator-metrics` | `join-total` | client-id | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | +| 18 | `consumer-coordinator-metrics` | `last-heartbeat-seconds-ago` | client-id | `kafka.consumer.last_heartbeat_seconds_ago` | The number of seconds since the last coordinator heartbeat was sent | `DOUBLE_OBSERVABLE_GAUGE` | +| 19 | `consumer-coordinator-metrics` | `last-rebalance-seconds-ago` | client-id | `kafka.consumer.last_rebalance_seconds_ago` | The number of seconds since the last successful rebalance event | `DOUBLE_OBSERVABLE_GAUGE` | +| 20 | `consumer-coordinator-metrics` | `partition-assigned-latency-avg` | client-id | `kafka.consumer.partition_assigned_latency_avg` | The average time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 21 | `consumer-coordinator-metrics` | `partition-assigned-latency-max` | client-id | `kafka.consumer.partition_assigned_latency_max` | The max time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 22 | `consumer-coordinator-metrics` | `partition-lost-latency-avg` | client-id | `kafka.consumer.partition_lost_latency_avg` | The average time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 23 | `consumer-coordinator-metrics` | `partition-lost-latency-max` | client-id | `kafka.consumer.partition_lost_latency_max` | The max time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 24 | `consumer-coordinator-metrics` | `partition-revoked-latency-avg` | client-id | `kafka.consumer.partition_revoked_latency_avg` | The average time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 25 | `consumer-coordinator-metrics` | `partition-revoked-latency-max` | client-id | `kafka.consumer.partition_revoked_latency_max` | The max time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 26 | `consumer-coordinator-metrics` | `rebalance-latency-avg` | client-id | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| 27 | `consumer-coordinator-metrics` | `rebalance-latency-max` | client-id | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| 28 | `consumer-coordinator-metrics` | `rebalance-latency-total` | client-id | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | +| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | client-id | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 30 | `consumer-coordinator-metrics` | `rebalance-total` | client-id | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | +| 31 | `consumer-coordinator-metrics` | `sync-rate` | client-id | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 32 | `consumer-coordinator-metrics` | `sync-time-avg` | client-id | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| 33 | `consumer-coordinator-metrics` | `sync-time-max` | client-id | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| 34 | `consumer-coordinator-metrics` | `sync-total` | client-id | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | +| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id,topic | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 37 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id,topic | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 38 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | client-id | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | client-id | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | client-id | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id,topic | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | client-id,topic | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | client-id | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 46 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | client-id | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | client-id | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 48 | `consumer-fetch-manager-metrics` | `fetch-total` | client-id | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | +| 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | client-id,topic,partition | | | | +| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id,topic | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id,topic | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 54 | `consumer-fetch-manager-metrics` | `records-lag` | client-id,topic,partition | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 55 | `consumer-fetch-manager-metrics` | `records-lag-avg` | client-id,topic,partition | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | client-id | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | client-id,topic,partition | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 58 | `consumer-fetch-manager-metrics` | `records-lead` | client-id,topic,partition | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 59 | `consumer-fetch-manager-metrics` | `records-lead-avg` | client-id,topic,partition | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 60 | `consumer-fetch-manager-metrics` | `records-lead-min` | client-id | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 61 | `consumer-fetch-manager-metrics` | `records-lead-min` | client-id,topic,partition | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 62 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | +| 63 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id,topic | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | +| 64 | `consumer-metrics` | `connection-close-rate` | client-id | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 65 | `consumer-metrics` | `connection-close-total` | client-id | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| 66 | `consumer-metrics` | `connection-count` | client-id | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| 67 | `consumer-metrics` | `connection-creation-rate` | client-id | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 68 | `consumer-metrics` | `connection-creation-total` | client-id | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| 69 | `consumer-metrics` | `failed-authentication-rate` | client-id | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 70 | `consumer-metrics` | `failed-authentication-total` | client-id | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 71 | `consumer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 72 | `consumer-metrics` | `failed-reauthentication-total` | client-id | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 73 | `consumer-metrics` | `incoming-byte-rate` | client-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 74 | `consumer-metrics` | `incoming-byte-total` | client-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 75 | `consumer-metrics` | `io-ratio` | client-id | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 76 | `consumer-metrics` | `io-time-ns-avg` | client-id | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 77 | `consumer-metrics` | `io-wait-ratio` | client-id | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 78 | `consumer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 79 | `consumer-metrics` | `io-waittime-total` | client-id | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| 80 | `consumer-metrics` | `iotime-total` | client-id | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| 81 | `consumer-metrics` | `last-poll-seconds-ago` | client-id | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | +| 82 | `consumer-metrics` | `network-io-rate` | client-id | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 83 | `consumer-metrics` | `network-io-total` | client-id | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 84 | `consumer-metrics` | `outgoing-byte-rate` | client-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 85 | `consumer-metrics` | `outgoing-byte-total` | client-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 86 | `consumer-metrics` | `poll-idle-ratio-avg` | client-id | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | +| 87 | `consumer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 88 | `consumer-metrics` | `reauthentication-latency-max` | client-id | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 89 | `consumer-metrics` | `request-rate` | client-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 90 | `consumer-metrics` | `request-size-avg` | client-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 91 | `consumer-metrics` | `request-size-max` | client-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 92 | `consumer-metrics` | `request-total` | client-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 93 | `consumer-metrics` | `response-rate` | client-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 94 | `consumer-metrics` | `response-total` | client-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 95 | `consumer-metrics` | `select-rate` | client-id | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 96 | `consumer-metrics` | `select-total` | client-id | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| 97 | `consumer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 98 | `consumer-metrics` | `successful-authentication-rate` | client-id | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 99 | `consumer-metrics` | `successful-authentication-total` | client-id | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 100 | `consumer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 101 | `consumer-metrics` | `successful-reauthentication-total` | client-id | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 102 | `consumer-metrics` | `time-between-poll-avg` | client-id | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| 103 | `consumer-metrics` | `time-between-poll-max` | client-id | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| 104 | `consumer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 105 | `consumer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 107 | `consumer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 108 | `consumer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | +| 109 | `consumer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | +| 110 | `consumer-node-metrics` | `request-rate` | client-id,node-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 111 | `consumer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 112 | `consumer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 113 | `consumer-node-metrics` | `request-total` | client-id,node-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 114 | `consumer-node-metrics` | `response-rate` | client-id,node-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 115 | `consumer-node-metrics` | `response-total` | client-id,node-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 116 | `kafka-metrics-count` | `count` | client-id | | | | +| 117 | `producer-metrics` | `batch-size-avg` | client-id | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 118 | `producer-metrics` | `batch-size-max` | client-id | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 119 | `producer-metrics` | `batch-split-rate` | client-id | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 120 | `producer-metrics` | `batch-split-total` | client-id | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | +| 121 | `producer-metrics` | `buffer-available-bytes` | client-id | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | +| 122 | `producer-metrics` | `buffer-exhausted-rate` | client-id | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 123 | `producer-metrics` | `buffer-exhausted-total` | client-id | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | +| 124 | `producer-metrics` | `buffer-total-bytes` | client-id | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | +| 125 | `producer-metrics` | `bufferpool-wait-ratio` | client-id | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 126 | `producer-metrics` | `bufferpool-wait-time-total` | client-id | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | +| 127 | `producer-metrics` | `compression-rate-avg` | client-id | `kafka.producer.compression_rate_avg` | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | +| 128 | `producer-metrics` | `connection-close-rate` | client-id | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 129 | `producer-metrics` | `connection-close-total` | client-id | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| 130 | `producer-metrics` | `connection-count` | client-id | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| 131 | `producer-metrics` | `connection-creation-rate` | client-id | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 132 | `producer-metrics` | `connection-creation-total` | client-id | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| 133 | `producer-metrics` | `failed-authentication-rate` | client-id | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 134 | `producer-metrics` | `failed-authentication-total` | client-id | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 135 | `producer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 136 | `producer-metrics` | `failed-reauthentication-total` | client-id | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 137 | `producer-metrics` | `incoming-byte-rate` | client-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 138 | `producer-metrics` | `incoming-byte-total` | client-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 139 | `producer-metrics` | `io-ratio` | client-id | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 140 | `producer-metrics` | `io-time-ns-avg` | client-id | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 141 | `producer-metrics` | `io-wait-ratio` | client-id | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 142 | `producer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 143 | `producer-metrics` | `io-waittime-total` | client-id | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| 144 | `producer-metrics` | `iotime-total` | client-id | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| 145 | `producer-metrics` | `metadata-age` | client-id | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | +| 146 | `producer-metrics` | `network-io-rate` | client-id | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 147 | `producer-metrics` | `network-io-total` | client-id | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 148 | `producer-metrics` | `outgoing-byte-rate` | client-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 149 | `producer-metrics` | `outgoing-byte-total` | client-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 150 | `producer-metrics` | `produce-throttle-time-avg` | client-id | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| 151 | `producer-metrics` | `produce-throttle-time-max` | client-id | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| 152 | `producer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 153 | `producer-metrics` | `reauthentication-latency-max` | client-id | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 154 | `producer-metrics` | `record-error-rate` | client-id | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 155 | `producer-metrics` | `record-error-total` | client-id | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | +| 156 | `producer-metrics` | `record-queue-time-avg` | client-id | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| 157 | `producer-metrics` | `record-queue-time-max` | client-id | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| 158 | `producer-metrics` | `record-retry-rate` | client-id | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 159 | `producer-metrics` | `record-retry-total` | client-id | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | +| 160 | `producer-metrics` | `record-send-rate` | client-id | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 161 | `producer-metrics` | `record-send-total` | client-id | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | +| 162 | `producer-metrics` | `record-size-avg` | client-id | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | +| 163 | `producer-metrics` | `record-size-max` | client-id | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | +| 164 | `producer-metrics` | `records-per-request-avg` | client-id | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 165 | `producer-metrics` | `request-latency-avg` | client-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 166 | `producer-metrics` | `request-latency-max` | client-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 167 | `producer-metrics` | `request-rate` | client-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 168 | `producer-metrics` | `request-size-avg` | client-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 169 | `producer-metrics` | `request-size-max` | client-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 170 | `producer-metrics` | `request-total` | client-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 171 | `producer-metrics` | `requests-in-flight` | client-id | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | +| 172 | `producer-metrics` | `response-rate` | client-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 173 | `producer-metrics` | `response-total` | client-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 174 | `producer-metrics` | `select-rate` | client-id | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 175 | `producer-metrics` | `select-total` | client-id | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| 176 | `producer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 177 | `producer-metrics` | `successful-authentication-rate` | client-id | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 178 | `producer-metrics` | `successful-authentication-total` | client-id | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 179 | `producer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 180 | `producer-metrics` | `successful-reauthentication-total` | client-id | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 181 | `producer-metrics` | `waiting-threads` | client-id | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | +| 182 | `producer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 183 | `producer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 184 | `producer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 185 | `producer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 186 | `producer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 187 | `producer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 188 | `producer-node-metrics` | `request-rate` | client-id,node-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 189 | `producer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 190 | `producer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 191 | `producer-node-metrics` | `request-total` | client-id,node-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 192 | `producer-node-metrics` | `response-rate` | client-id,node-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 193 | `producer-node-metrics` | `response-total` | client-id,node-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 194 | `producer-topic-metrics` | `byte-rate` | client-id,topic | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 195 | `producer-topic-metrics` | `byte-total` | client-id,topic | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | +| 196 | `producer-topic-metrics` | `compression-rate` | client-id,topic | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | +| 197 | `producer-topic-metrics` | `record-error-rate` | client-id,topic | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 198 | `producer-topic-metrics` | `record-error-total` | client-id,topic | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | +| 199 | `producer-topic-metrics` | `record-retry-rate` | client-id,topic | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 200 | `producer-topic-metrics` | `record-retry-total` | client-id,topic | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | +| 201 | `producer-topic-metrics` | `record-send-rate` | client-id,topic | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 202 | `producer-topic-metrics` | `record-send-total` | client-id,topic | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index c54818158edb..041e2c0cebaf 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -64,15 +64,13 @@ static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) if (!matchingGroup.isPresent()) { return null; } - String instrumentName = "kafka." + matchingGroup.get() + "." + kafkaMetricId.getName(); + String instrumentName = + "kafka." + matchingGroup.get() + "." + kafkaMetricId.getName().replace("-", "_"); String description = descriptionCache.computeIfAbsent(instrumentName, s -> kafkaMetricId.getDescription()); String instrumentType = - measureableToInstrumentType.entrySet().stream() - .filter(entry -> entry.getKey().equals(kafkaMetricId.getMeasureable())) - .findFirst() - .map(Map.Entry::getValue) - .orElse(INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + measureableToInstrumentType.getOrDefault( + kafkaMetricId.getMeasureable(), INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); switch (instrumentType) { case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index fec1fb8bfd56..37a1aa5b0b05 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -70,16 +70,7 @@ public static void setOpenTelemetry(OpenTelemetry openTelemetry) { */ static void resetForTest() { meter = null; - for (Iterator> it = - instrumentMap.entrySet().iterator(); - it.hasNext(); ) { - try { - it.next().getValue().close(); - it.remove(); - } catch (Exception e) { - throw new IllegalStateException("Error occurred closing instrument", e); - } - } + closeAllInstruments(); } /** @@ -112,12 +103,12 @@ static void printMappingTable() { String.format( "| %s | %s | %s | %s | %s | %s | %s |%n", count, - group, - kafkaMetricId.getName(), + "`" + group + "`", + "`" + kafkaMetricId.getName() + "`", String.join(",", kafkaMetricId.getAttributeKeys()), - descriptor.map(InstrumentDescriptor::getName).orElse(""), + descriptor.map(i -> "`" + i.getName() + "`").orElse(""), descriptor.map(InstrumentDescriptor::getDescription).orElse(""), - descriptor.map(InstrumentDescriptor::getInstrumentType).orElse(""))); + descriptor.map(i -> "`" + i.getInstrumentType() + "`").orElse(""))); count++; } } @@ -132,37 +123,35 @@ public void init(List metrics) { @Override public void metricChange(KafkaMetric metric) { KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); - seenMetrics.add(KafkaMetricId.create(metric)); + seenMetrics.add(kafkaMetricId); Meter currentMeter = meter; if (currentMeter == null) { - logger.log(Level.FINEST, "Metric changed but meter not set: " + kafkaMetricId); + logger.log(Level.FINEST, "Metric changed but meter not set: {0}", kafkaMetricId); return; } InstrumentDescriptor instrumentDescriptor = KafkaMetricRegistry.getInstrumentDescriptor(kafkaMetricId); if (instrumentDescriptor == null) { - logger.log(Level.FINEST, "Metric changed but cannot map to instrument: " + kafkaMetricId); + logger.log(Level.FINEST, "Metric changed but cannot map to instrument: {0}", kafkaMetricId); return; } - AttributesBuilder attributesBuilder = Attributes.builder(); - metric.metricName().tags().forEach(attributesBuilder::put); RegisteredInstrument registeredInstrument = - RegisteredInstrument.create(kafkaMetricId, attributesBuilder.build()); + RegisteredInstrument.create(kafkaMetricId, toAttributes(metric)); instrumentMap.compute( registeredInstrument, (registeredInstrument1, autoCloseable) -> { if (autoCloseable != null) { - logger.log(Level.FINEST, "Replacing instrument " + registeredInstrument1); + logger.log(Level.FINEST, "Replacing instrument {0}", registeredInstrument1); try { autoCloseable.close(); } catch (Exception e) { logger.log(Level.WARNING, "An error occurred closing instrument", e); } } else { - logger.log(Level.FINEST, "Adding instrument " + registeredInstrument1); + logger.log(Level.FINEST, "Adding instrument {0}", registeredInstrument1); } return createObservable( currentMeter, registeredInstrument1, instrumentDescriptor, metric); @@ -214,11 +203,34 @@ public void metricRemoval(KafkaMetric metric) { KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); seenMetrics.add(kafkaMetricId); logger.log(Level.FINEST, "Metric removed: " + kafkaMetricId); - instrumentMap.remove(RegisteredInstrument.create(kafkaMetricId, toAttributes(metric))); + AutoCloseable observable = + instrumentMap.remove(RegisteredInstrument.create(kafkaMetricId, toAttributes(metric))); + if (observable != null) { + closeInstrument(observable); + } } @Override - public void close() {} + public void close() { + closeAllInstruments(); + } + + static void closeAllInstruments() { + for (Iterator> it = + instrumentMap.entrySet().iterator(); + it.hasNext(); ) { + closeInstrument(it.next().getValue()); + it.remove(); + } + } + + private static void closeInstrument(AutoCloseable observable) { + try { + observable.close(); + } catch (Exception e) { + throw new IllegalStateException("Error occurred closing instrument", e); + } + } @Override public void configure(Map configs) {} From 11702a18d2110b1b43be2f98422cd65a2824bb37 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 21 Jun 2022 18:36:42 -0500 Subject: [PATCH 07/17] Map rate measureables to gauges instead of up down counters --- .../kafka-clients-common/library/README.md | 114 +++++++++--------- .../kafkaclients/InstrumentDescriptor.java | 6 - .../kafkaclients/KafkaMetricRegistry.java | 5 +- .../OpenTelemetryKafkaMetrics.java | 7 -- 4 files changed, 58 insertions(+), 74 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md index 832f550f217d..aa754a21a6a2 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md @@ -28,14 +28,14 @@ collected. | 4 | `consumer-coordinator-metrics` | `assigned-partitions` | client-id | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | | 5 | `consumer-coordinator-metrics` | `commit-latency-avg` | client-id | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | | 6 | `consumer-coordinator-metrics` | `commit-latency-max` | client-id | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | -| 7 | `consumer-coordinator-metrics` | `commit-rate` | client-id | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 7 | `consumer-coordinator-metrics` | `commit-rate` | client-id | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_GAUGE` | | 8 | `consumer-coordinator-metrics` | `commit-total` | client-id | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | -| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | client-id | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | client-id | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_GAUGE` | | 10 | `consumer-coordinator-metrics` | `failed-rebalance-total` | client-id | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | -| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | client-id | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | client-id | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_GAUGE` | | 12 | `consumer-coordinator-metrics` | `heartbeat-response-time-max` | client-id | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | | 13 | `consumer-coordinator-metrics` | `heartbeat-total` | client-id | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | -| 14 | `consumer-coordinator-metrics` | `join-rate` | client-id | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 14 | `consumer-coordinator-metrics` | `join-rate` | client-id | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_GAUGE` | | 15 | `consumer-coordinator-metrics` | `join-time-avg` | client-id | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | | 16 | `consumer-coordinator-metrics` | `join-time-max` | client-id | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | | 17 | `consumer-coordinator-metrics` | `join-total` | client-id | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | @@ -50,19 +50,19 @@ collected. | 26 | `consumer-coordinator-metrics` | `rebalance-latency-avg` | client-id | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | | 27 | `consumer-coordinator-metrics` | `rebalance-latency-max` | client-id | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | | 28 | `consumer-coordinator-metrics` | `rebalance-latency-total` | client-id | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | -| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | client-id | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | client-id | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | | 30 | `consumer-coordinator-metrics` | `rebalance-total` | client-id | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | -| 31 | `consumer-coordinator-metrics` | `sync-rate` | client-id | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 31 | `consumer-coordinator-metrics` | `sync-rate` | client-id | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_GAUGE` | | 32 | `consumer-coordinator-metrics` | `sync-time-avg` | client-id | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | | 33 | `consumer-coordinator-metrics` | `sync-time-max` | client-id | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | | 34 | `consumer-coordinator-metrics` | `sync-total` | client-id | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | -| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id,topic | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | -| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id,topic | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | | 37 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id,topic | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | | 38 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | | 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | client-id | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | | 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | client-id | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | client-id | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | client-id | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | | 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | | 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id,topic | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | | 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | client-id,topic | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | @@ -71,8 +71,8 @@ collected. | 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | client-id | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | | 48 | `consumer-fetch-manager-metrics` | `fetch-total` | client-id | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | | 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | client-id,topic,partition | | | | -| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | -| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id,topic | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id,topic | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | | 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | | 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id,topic | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | | 54 | `consumer-fetch-manager-metrics` | `records-lag` | client-id,topic,partition | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | @@ -85,142 +85,142 @@ collected. | 61 | `consumer-fetch-manager-metrics` | `records-lead-min` | client-id,topic,partition | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | | 62 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | | 63 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id,topic | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | -| 64 | `consumer-metrics` | `connection-close-rate` | client-id | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 64 | `consumer-metrics` | `connection-close-rate` | client-id | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | | 65 | `consumer-metrics` | `connection-close-total` | client-id | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | | 66 | `consumer-metrics` | `connection-count` | client-id | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 67 | `consumer-metrics` | `connection-creation-rate` | client-id | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 67 | `consumer-metrics` | `connection-creation-rate` | client-id | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | | 68 | `consumer-metrics` | `connection-creation-total` | client-id | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 69 | `consumer-metrics` | `failed-authentication-rate` | client-id | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 69 | `consumer-metrics` | `failed-authentication-rate` | client-id | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | | 70 | `consumer-metrics` | `failed-authentication-total` | client-id | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 71 | `consumer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 71 | `consumer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 72 | `consumer-metrics` | `failed-reauthentication-total` | client-id | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 73 | `consumer-metrics` | `incoming-byte-rate` | client-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 73 | `consumer-metrics` | `incoming-byte-rate` | client-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | | 74 | `consumer-metrics` | `incoming-byte-total` | client-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 75 | `consumer-metrics` | `io-ratio` | client-id | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 75 | `consumer-metrics` | `io-ratio` | client-id | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | | 76 | `consumer-metrics` | `io-time-ns-avg` | client-id | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 77 | `consumer-metrics` | `io-wait-ratio` | client-id | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 77 | `consumer-metrics` | `io-wait-ratio` | client-id | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | | 78 | `consumer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | | 79 | `consumer-metrics` | `io-waittime-total` | client-id | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | | 80 | `consumer-metrics` | `iotime-total` | client-id | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | | 81 | `consumer-metrics` | `last-poll-seconds-ago` | client-id | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | -| 82 | `consumer-metrics` | `network-io-rate` | client-id | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 82 | `consumer-metrics` | `network-io-rate` | client-id | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 83 | `consumer-metrics` | `network-io-total` | client-id | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 84 | `consumer-metrics` | `outgoing-byte-rate` | client-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 84 | `consumer-metrics` | `outgoing-byte-rate` | client-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | | 85 | `consumer-metrics` | `outgoing-byte-total` | client-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | | 86 | `consumer-metrics` | `poll-idle-ratio-avg` | client-id | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | | 87 | `consumer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | | 88 | `consumer-metrics` | `reauthentication-latency-max` | client-id | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 89 | `consumer-metrics` | `request-rate` | client-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 89 | `consumer-metrics` | `request-rate` | client-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | | 90 | `consumer-metrics` | `request-size-avg` | client-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 91 | `consumer-metrics` | `request-size-max` | client-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 92 | `consumer-metrics` | `request-total` | client-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 93 | `consumer-metrics` | `response-rate` | client-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 93 | `consumer-metrics` | `response-rate` | client-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | | 94 | `consumer-metrics` | `response-total` | client-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 95 | `consumer-metrics` | `select-rate` | client-id | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 95 | `consumer-metrics` | `select-rate` | client-id | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | | 96 | `consumer-metrics` | `select-total` | client-id | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | | 97 | `consumer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 98 | `consumer-metrics` | `successful-authentication-rate` | client-id | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 98 | `consumer-metrics` | `successful-authentication-rate` | client-id | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | | 99 | `consumer-metrics` | `successful-authentication-total` | client-id | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 100 | `consumer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 100 | `consumer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 101 | `consumer-metrics` | `successful-reauthentication-total` | client-id | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | | 102 | `consumer-metrics` | `time-between-poll-avg` | client-id | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | | 103 | `consumer-metrics` | `time-between-poll-max` | client-id | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | -| 104 | `consumer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 104 | `consumer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | | 105 | `consumer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | | 107 | `consumer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | | 108 | `consumer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | | 109 | `consumer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | -| 110 | `consumer-node-metrics` | `request-rate` | client-id,node-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 110 | `consumer-node-metrics` | `request-rate` | client-id,node-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | | 111 | `consumer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 112 | `consumer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 113 | `consumer-node-metrics` | `request-total` | client-id,node-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 114 | `consumer-node-metrics` | `response-rate` | client-id,node-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 114 | `consumer-node-metrics` | `response-rate` | client-id,node-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | | 115 | `consumer-node-metrics` | `response-total` | client-id,node-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | | 116 | `kafka-metrics-count` | `count` | client-id | | | | | 117 | `producer-metrics` | `batch-size-avg` | client-id | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | | 118 | `producer-metrics` | `batch-size-max` | client-id | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 119 | `producer-metrics` | `batch-split-rate` | client-id | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 119 | `producer-metrics` | `batch-split-rate` | client-id | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_GAUGE` | | 120 | `producer-metrics` | `batch-split-total` | client-id | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | | 121 | `producer-metrics` | `buffer-available-bytes` | client-id | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | -| 122 | `producer-metrics` | `buffer-exhausted-rate` | client-id | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 122 | `producer-metrics` | `buffer-exhausted-rate` | client-id | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_GAUGE` | | 123 | `producer-metrics` | `buffer-exhausted-total` | client-id | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | | 124 | `producer-metrics` | `buffer-total-bytes` | client-id | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | -| 125 | `producer-metrics` | `bufferpool-wait-ratio` | client-id | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 125 | `producer-metrics` | `bufferpool-wait-ratio` | client-id | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_GAUGE` | | 126 | `producer-metrics` | `bufferpool-wait-time-total` | client-id | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | | 127 | `producer-metrics` | `compression-rate-avg` | client-id | `kafka.producer.compression_rate_avg` | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 128 | `producer-metrics` | `connection-close-rate` | client-id | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 128 | `producer-metrics` | `connection-close-rate` | client-id | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | | 129 | `producer-metrics` | `connection-close-total` | client-id | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | | 130 | `producer-metrics` | `connection-count` | client-id | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 131 | `producer-metrics` | `connection-creation-rate` | client-id | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 131 | `producer-metrics` | `connection-creation-rate` | client-id | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | | 132 | `producer-metrics` | `connection-creation-total` | client-id | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 133 | `producer-metrics` | `failed-authentication-rate` | client-id | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 133 | `producer-metrics` | `failed-authentication-rate` | client-id | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | | 134 | `producer-metrics` | `failed-authentication-total` | client-id | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 135 | `producer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 135 | `producer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 136 | `producer-metrics` | `failed-reauthentication-total` | client-id | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 137 | `producer-metrics` | `incoming-byte-rate` | client-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 137 | `producer-metrics` | `incoming-byte-rate` | client-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | | 138 | `producer-metrics` | `incoming-byte-total` | client-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 139 | `producer-metrics` | `io-ratio` | client-id | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 139 | `producer-metrics` | `io-ratio` | client-id | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | | 140 | `producer-metrics` | `io-time-ns-avg` | client-id | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 141 | `producer-metrics` | `io-wait-ratio` | client-id | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 141 | `producer-metrics` | `io-wait-ratio` | client-id | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | | 142 | `producer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | | 143 | `producer-metrics` | `io-waittime-total` | client-id | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | | 144 | `producer-metrics` | `iotime-total` | client-id | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | | 145 | `producer-metrics` | `metadata-age` | client-id | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | -| 146 | `producer-metrics` | `network-io-rate` | client-id | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 146 | `producer-metrics` | `network-io-rate` | client-id | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 147 | `producer-metrics` | `network-io-total` | client-id | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 148 | `producer-metrics` | `outgoing-byte-rate` | client-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 148 | `producer-metrics` | `outgoing-byte-rate` | client-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | | 149 | `producer-metrics` | `outgoing-byte-total` | client-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | | 150 | `producer-metrics` | `produce-throttle-time-avg` | client-id | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | | 151 | `producer-metrics` | `produce-throttle-time-max` | client-id | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | | 152 | `producer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | | 153 | `producer-metrics` | `reauthentication-latency-max` | client-id | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 154 | `producer-metrics` | `record-error-rate` | client-id | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 154 | `producer-metrics` | `record-error-rate` | client-id | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | | 155 | `producer-metrics` | `record-error-total` | client-id | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | | 156 | `producer-metrics` | `record-queue-time-avg` | client-id | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | | 157 | `producer-metrics` | `record-queue-time-max` | client-id | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | -| 158 | `producer-metrics` | `record-retry-rate` | client-id | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 158 | `producer-metrics` | `record-retry-rate` | client-id | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | | 159 | `producer-metrics` | `record-retry-total` | client-id | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | -| 160 | `producer-metrics` | `record-send-rate` | client-id | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 160 | `producer-metrics` | `record-send-rate` | client-id | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | | 161 | `producer-metrics` | `record-send-total` | client-id | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | | 162 | `producer-metrics` | `record-size-avg` | client-id | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | | 163 | `producer-metrics` | `record-size-max` | client-id | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | | 164 | `producer-metrics` | `records-per-request-avg` | client-id | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | | 165 | `producer-metrics` | `request-latency-avg` | client-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | | 166 | `producer-metrics` | `request-latency-max` | client-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 167 | `producer-metrics` | `request-rate` | client-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 167 | `producer-metrics` | `request-rate` | client-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | | 168 | `producer-metrics` | `request-size-avg` | client-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 169 | `producer-metrics` | `request-size-max` | client-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 170 | `producer-metrics` | `request-total` | client-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | | 171 | `producer-metrics` | `requests-in-flight` | client-id | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | -| 172 | `producer-metrics` | `response-rate` | client-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 172 | `producer-metrics` | `response-rate` | client-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | | 173 | `producer-metrics` | `response-total` | client-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 174 | `producer-metrics` | `select-rate` | client-id | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 174 | `producer-metrics` | `select-rate` | client-id | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | | 175 | `producer-metrics` | `select-total` | client-id | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | | 176 | `producer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 177 | `producer-metrics` | `successful-authentication-rate` | client-id | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 177 | `producer-metrics` | `successful-authentication-rate` | client-id | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | | 178 | `producer-metrics` | `successful-authentication-total` | client-id | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 179 | `producer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 179 | `producer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | | 180 | `producer-metrics` | `successful-reauthentication-total` | client-id | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | | 181 | `producer-metrics` | `waiting-threads` | client-id | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | -| 182 | `producer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 182 | `producer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | | 183 | `producer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 184 | `producer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 184 | `producer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | | 185 | `producer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | | 186 | `producer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | | 187 | `producer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 188 | `producer-node-metrics` | `request-rate` | client-id,node-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 188 | `producer-node-metrics` | `request-rate` | client-id,node-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | | 189 | `producer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 190 | `producer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | | 191 | `producer-node-metrics` | `request-total` | client-id,node-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 192 | `producer-node-metrics` | `response-rate` | client-id,node-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 192 | `producer-node-metrics` | `response-rate` | client-id,node-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | | 193 | `producer-node-metrics` | `response-total` | client-id,node-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 194 | `producer-topic-metrics` | `byte-rate` | client-id,topic | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 194 | `producer-topic-metrics` | `byte-rate` | client-id,topic | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | | 195 | `producer-topic-metrics` | `byte-total` | client-id,topic | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | | 196 | `producer-topic-metrics` | `compression-rate` | client-id,topic | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 197 | `producer-topic-metrics` | `record-error-rate` | client-id,topic | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 197 | `producer-topic-metrics` | `record-error-rate` | client-id,topic | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | | 198 | `producer-topic-metrics` | `record-error-total` | client-id,topic | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | -| 199 | `producer-topic-metrics` | `record-retry-rate` | client-id,topic | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 199 | `producer-topic-metrics` | `record-retry-rate` | client-id,topic | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | | 200 | `producer-topic-metrics` | `record-retry-total` | client-id,topic | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | -| 201 | `producer-topic-metrics` | `record-send-rate` | client-id,topic | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_UP_DOWN_COUNTER` | +| 201 | `producer-topic-metrics` | `record-send-rate` | client-id,topic | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | | 202 | `producer-topic-metrics` | `record-send-total` | client-id,topic | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java index d69b0b0770a7..7330d57b9d15 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java @@ -12,8 +12,6 @@ abstract class InstrumentDescriptor { static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE = "DOUBLE_OBSERVABLE_GAUGE"; - static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER = - "DOUBLE_OBSERVABLE_UP_DOWN_COUNTER"; static final String INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER = "DOUBLE_OBSERVABLE_COUNTER"; abstract String getName(); @@ -32,8 +30,4 @@ static InstrumentDescriptor createDoubleCounter(String name, String description) name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER); } - static InstrumentDescriptor createDoubleUpDownCounter(String name, String description) { - return new AutoValue_InstrumentDescriptor( - name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER); - } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index 041e2c0cebaf..76e6474a8e31 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -7,7 +7,6 @@ import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; -import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER; import java.util.Arrays; import java.util.HashMap; @@ -29,7 +28,7 @@ class KafkaMetricRegistry { Map classNameToType = new HashMap<>(); classNameToType.put( "org.apache.kafka.common.metrics.stats.Rate", - INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER); + INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); classNameToType.put( "org.apache.kafka.common.metrics.stats.Avg", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); classNameToType.put( @@ -77,8 +76,6 @@ static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) return InstrumentDescriptor.createDoubleGauge(instrumentName, description); case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: return InstrumentDescriptor.createDoubleCounter(instrumentName, description); - case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER: - return InstrumentDescriptor.createDoubleUpDownCounter(instrumentName, description); default: // Continue below to throw } throw new IllegalStateException("Unrecognized instrument type. This is a bug."); diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 37a1aa5b0b05..2b2032cde5bd 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -7,7 +7,6 @@ import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; -import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER; import static java.lang.System.lineSeparator; import static java.util.Comparator.comparing; import static java.util.stream.Collectors.groupingBy; @@ -178,12 +177,6 @@ private static AutoCloseable createObservable( .setDescription(instrumentDescriptor.getDescription()) .ofDoubles() .buildWithCallback(callback); - case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_UP_DOWN_COUNTER: - return meter - .upDownCounterBuilder(instrumentDescriptor.getName()) - .setDescription(instrumentDescriptor.getDescription()) - .ofDoubles() - .buildWithCallback(callback); default: // Continue below to throw } // TODO: add support for other instrument types and value types as needed for new instruments From 4965b226d1ccac6f62312262b4f0eafba6e5d136 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Wed, 22 Jun 2022 13:57:12 -0500 Subject: [PATCH 08/17] Spotless, quote attributes, log placeholder --- .../instrumentation/kafkaclients/InstrumentDescriptor.java | 1 - .../instrumentation/kafkaclients/KafkaMetricRegistry.java | 3 +-- .../kafkaclients/OpenTelemetryKafkaMetrics.java | 7 +++++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java index 7330d57b9d15..61a16dd222a1 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java @@ -29,5 +29,4 @@ static InstrumentDescriptor createDoubleCounter(String name, String description) return new AutoValue_InstrumentDescriptor( name, description, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER); } - } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index 76e6474a8e31..c0c052a82768 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -27,8 +27,7 @@ class KafkaMetricRegistry { static { Map classNameToType = new HashMap<>(); classNameToType.put( - "org.apache.kafka.common.metrics.stats.Rate", - INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + "org.apache.kafka.common.metrics.stats.Rate", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); classNameToType.put( "org.apache.kafka.common.metrics.stats.Avg", INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); classNameToType.put( diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 2b2032cde5bd..1174eb26b905 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -10,6 +10,7 @@ import static java.lang.System.lineSeparator; import static java.util.Comparator.comparing; import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.joining; import static java.util.stream.Collectors.toList; import io.opentelemetry.api.OpenTelemetry; @@ -104,7 +105,9 @@ static void printMappingTable() { count, "`" + group + "`", "`" + kafkaMetricId.getName() + "`", - String.join(",", kafkaMetricId.getAttributeKeys()), + kafkaMetricId.getAttributeKeys().stream() + .map(key -> "`" + key + "`") + .collect(joining(",")), descriptor.map(i -> "`" + i.getName() + "`").orElse(""), descriptor.map(InstrumentDescriptor::getDescription).orElse(""), descriptor.map(i -> "`" + i.getInstrumentType() + "`").orElse(""))); @@ -195,7 +198,7 @@ private static Attributes toAttributes(KafkaMetric kafkaMetric) { public void metricRemoval(KafkaMetric metric) { KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); seenMetrics.add(kafkaMetricId); - logger.log(Level.FINEST, "Metric removed: " + kafkaMetricId); + logger.log(Level.FINEST, "Metric removed: {0}", kafkaMetricId); AutoCloseable observable = instrumentMap.remove(RegisteredInstrument.create(kafkaMetricId, toAttributes(metric))); if (observable != null) { From 049d80f8fd9349a97dc42648935a7e19360e299d Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Wed, 22 Jun 2022 17:17:36 -0500 Subject: [PATCH 09/17] Move metric table printing to test, only retain most granular attribute set --- .../OpenTelemetryKafkaMetricsTest.java | 214 +++++++-- .../kafka-clients-common/library/README.md | 411 +++++++++--------- .../kafkaclients/KafkaMetricRegistry.java | 57 ++- .../OpenTelemetryKafkaMetrics.java | 165 ++----- .../kafkaclients/RegisteredInstrument.java | 21 - .../kafkaclients/RegisteredObservable.java | 38 ++ 6 files changed, 513 insertions(+), 393 deletions(-) delete mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index b7dcf128efa8..a962763041c1 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -5,31 +5,47 @@ package io.opentelemetry.instrumentation.kafkaclients; +import static java.lang.System.lineSeparator; +import static java.util.Comparator.comparing; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.joining; +import static java.util.stream.Collectors.toList; import static java.util.stream.Collectors.toSet; import static org.assertj.core.api.Assertions.assertThat; +import io.opentelemetry.api.common.AttributeKey; import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension; import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension; import io.opentelemetry.sdk.metrics.data.MetricData; +import io.opentelemetry.sdk.metrics.data.PointData; import java.nio.charset.StandardCharsets; import java.time.Duration; import java.time.Instant; import java.util.Arrays; +import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Objects; +import java.util.Optional; import java.util.Random; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.metrics.KafkaMetric; +import org.apache.kafka.common.metrics.MetricsReporter; import org.apache.kafka.common.serialization.ByteArrayDeserializer; import org.apache.kafka.common.serialization.ByteArraySerializer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.RegisterExtension; +import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.testcontainers.containers.KafkaContainer; import org.testcontainers.containers.output.Slf4jLogConsumer; @@ -41,24 +57,65 @@ @Testcontainers abstract class OpenTelemetryKafkaMetricsTest { + private static final Logger logger = LoggerFactory.getLogger(OpenTelemetryKafkaMetricsTest.class); + private static final List TOPICS = Arrays.asList("foo", "bar", "baz", "qux"); private static final Random RANDOM = new Random(); @Container KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3")) - .withLogConsumer( - new Slf4jLogConsumer(LoggerFactory.getLogger(OpenTelemetryKafkaMetricsTest.class))) + .withLogConsumer(new Slf4jLogConsumer(logger)) .waitingFor(Wait.forLogMessage(".*started \\(kafka.server.KafkaServer\\).*", 1)) .withStartupTimeout(Duration.ofMinutes(1)); @RegisterExtension static final InstrumentationExtension testing = LibraryInstrumentationExtension.create(); - @Test - void observeMetrics() { + private KafkaProducer producer; + private KafkaConsumer consumer; + + @BeforeEach + void setup() { OpenTelemetryKafkaMetrics.setOpenTelemetry(testing.getOpenTelemetry()); + String metricReporters = + OpenTelemetryKafkaMetrics.class.getName() + "," + TestMetricsReporter.class.getName(); + + Map producerConfig = new HashMap<>(); + // Register OpenTelemetryKafkaMetrics as reporter + producerConfig.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, metricReporters); + producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); + producerConfig.put(ProducerConfig.CLIENT_ID_CONFIG, "sample-client-id"); + producerConfig.put( + ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); + producerConfig.put( + ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); + producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip"); + producer = new KafkaProducer<>(producerConfig); + + Map consumerConfig = new HashMap<>(); + // Register OpenTelemetryKafkaMetrics as reporter + consumerConfig.put(ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG, metricReporters); + consumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); + consumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-group"); + consumerConfig.put( + ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); + consumerConfig.put( + ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); + consumerConfig.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2); + consumer = new KafkaConsumer<>(consumerConfig); + } + + @AfterEach + void tearDown() { + OpenTelemetryKafkaMetrics.resetForTest(); + producer.close(); + consumer.close(); + } + + @Test + void observeMetrics() { produceRecords(); consumeRecords(); @@ -253,56 +310,129 @@ void observeMetrics() { "kafka.producer.record_send_rate", "kafka.producer.record_send_total")); - Set metricNames = testing.metrics().stream().map(MetricData::getName).collect(toSet()); - + List metrics = testing.metrics(); + Set metricNames = metrics.stream().map(MetricData::getName).collect(toSet()); assertThat(metricNames).containsAll(expectedMetricNames); + assertThat(metrics) + .allSatisfy( + metricData -> { + Set expectedKeys = + metricData.getData().getPoints().stream() + .findFirst() + .map( + point -> + point.getAttributes().asMap().keySet().stream() + .map(AttributeKey::getKey) + .collect(toSet())) + .orElse(Collections.emptySet()); + assertThat(metricData.getData().getPoints()) + .extracting(PointData::getAttributes) + .extracting( + attributes -> + attributes.asMap().keySet().stream() + .map(AttributeKey::getKey) + .collect(toSet())) + .allSatisfy(attributeKeys -> assertThat(attributeKeys).isEqualTo(expectedKeys)); + }); + // Print mapping table - OpenTelemetryKafkaMetrics.printMappingTable(); + printMappingTable(); } void produceRecords() { - Map config = new HashMap<>(); - // Register OpenTelemetryKafkaMetrics as reporter - config.put( - ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); - config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); - config.put(ProducerConfig.CLIENT_ID_CONFIG, "sample-client-id"); - config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); - config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName()); - config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip"); - - try (KafkaProducer producer = new KafkaProducer<>(config)) { - for (int i = 0; i < 100; i++) { - producer.send( - new ProducerRecord<>( - TOPICS.get(RANDOM.nextInt(TOPICS.size())), - 0, - System.currentTimeMillis(), - "key".getBytes(StandardCharsets.UTF_8), - "value".getBytes(StandardCharsets.UTF_8))); - } + for (int i = 0; i < 100; i++) { + producer.send( + new ProducerRecord<>( + TOPICS.get(RANDOM.nextInt(TOPICS.size())), + 0, + System.currentTimeMillis(), + "key".getBytes(StandardCharsets.UTF_8), + "value".getBytes(StandardCharsets.UTF_8))); } } void consumeRecords() { - Map config = new HashMap<>(); - // Register OpenTelemetryKafkaMetrics as reporter - config.put( - ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName()); - config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); - config.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-group"); - config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); - config.put( - ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName()); - config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2); + consumer.subscribe(TOPICS); + Instant stopTime = Instant.now().plusSeconds(10); + while (Instant.now().isBefore(stopTime)) { + consumer.poll(1000); + } + } - try (KafkaConsumer consumer = new KafkaConsumer<>(config)) { - consumer.subscribe(TOPICS); - Instant stopTime = Instant.now().plusSeconds(10); - while (Instant.now().isBefore(stopTime)) { - consumer.poll(1000); + /** + * Print a table mapping kafka metrics to equivalent OpenTelemetry metrics, in markdown format. + */ + private static void printMappingTable() { + StringBuilder sb = new StringBuilder(); + // Append table headers + sb.append( + "| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type |") + .append(lineSeparator()) + .append( + "|---|--------------|-------------|----------------|-----------------|------------------------|-----------------|") + .append(lineSeparator()); + Map> kafkaMetricsByGroup = + TestMetricsReporter.seenMetrics.keySet().stream() + .collect(groupingBy(KafkaMetricId::getGroup)); + List registeredObservables = + OpenTelemetryKafkaMetrics.getRegisteredObservables(); + int count = 1; + // Iterate through groups in alpha order + for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { + List kafkaMetricIds = + kafkaMetricsByGroup.get(group).stream() + .sorted(comparing(KafkaMetricId::getName)) + .collect(toList()); + // Iterate through metrics in alpha order by name + for (KafkaMetricId kafkaMetricId : kafkaMetricIds) { + KafkaMetric kafkaMetric = + Objects.requireNonNull(TestMetricsReporter.seenMetrics.get(kafkaMetricId)); + Optional descriptor = + registeredObservables.stream() + .filter(registeredObservable -> registeredObservable.matches(kafkaMetric)) + .findAny() + .map(RegisteredObservable::getInstrumentDescriptor); + // Append table row + sb.append( + String.format( + "| %s | %s | %s | %s | %s | %s | %s |%n", + count, + "`" + group + "`", + "`" + kafkaMetricId.getName() + "`", + kafkaMetricId.getAttributeKeys().stream() + .map(key -> "`" + key + "`") + .collect(joining(",")), + descriptor.map(i -> "`" + i.getName() + "`").orElse(""), + descriptor.map(InstrumentDescriptor::getDescription).orElse(""), + descriptor.map(i -> "`" + i.getInstrumentType() + "`").orElse(""))); + count++; } } + logger.info("Mapping table" + System.lineSeparator() + sb); + } + + public static class TestMetricsReporter implements MetricsReporter { + + private static final Map seenMetrics = new ConcurrentHashMap<>(); + + @Override + public void init(List list) { + list.forEach(this::metricChange); + } + + @Override + public void metricChange(KafkaMetric kafkaMetric) { + seenMetrics.put(KafkaMetricId.create(kafkaMetric), kafkaMetric); + } + + @Override + public void metricRemoval(KafkaMetric kafkaMetric) {} + + @Override + public void close() {} + + @Override + public void configure(Map map) {} } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md index aa754a21a6a2..ad2d004f8f43 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md @@ -15,6 +15,13 @@ config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString( try (KafkaProducer producer = new KafkaProducer<>(config)) { ... } ``` +Note: Kafka reports several metrics at multiple attribute granularities. For +example, `records-consumed-total` is reported with attribute key `[client-id]` +and `[client-id, topic]`. If you analyze the sum of records consumed, ignoring dimensions, backends +are likely to double count. To alleviate this, `OpenTelemetryKafkaMetrics` detects this +scenario and only records the most granular set of attributes available. In the case +of `records-consumed-total`, it reports `[client-id, topic]` and ignores `[client-id]`. + The following table shows the full set of metrics exposed by the kafka client, and the corresponding OpenTelemetry metric each maps to (if available). Empty values in the Instrument Name, Instrument Description, etc column indicates there is no registered mapping for the metric and data is NOT @@ -22,205 +29,205 @@ collected. | # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | |---|--------------|-------------|----------------|-----------------|------------------------|-----------------| -| 1 | `app-info` | `commit-id` | client-id | | | | -| 2 | `app-info` | `start-time-ms` | client-id | | | | -| 3 | `app-info` | `version` | client-id | | | | -| 4 | `consumer-coordinator-metrics` | `assigned-partitions` | client-id | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | -| 5 | `consumer-coordinator-metrics` | `commit-latency-avg` | client-id | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | -| 6 | `consumer-coordinator-metrics` | `commit-latency-max` | client-id | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | -| 7 | `consumer-coordinator-metrics` | `commit-rate` | client-id | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 8 | `consumer-coordinator-metrics` | `commit-total` | client-id | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | -| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | client-id | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_GAUGE` | -| 10 | `consumer-coordinator-metrics` | `failed-rebalance-total` | client-id | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | -| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | client-id | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 12 | `consumer-coordinator-metrics` | `heartbeat-response-time-max` | client-id | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | -| 13 | `consumer-coordinator-metrics` | `heartbeat-total` | client-id | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | -| 14 | `consumer-coordinator-metrics` | `join-rate` | client-id | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 15 | `consumer-coordinator-metrics` | `join-time-avg` | client-id | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | -| 16 | `consumer-coordinator-metrics` | `join-time-max` | client-id | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | -| 17 | `consumer-coordinator-metrics` | `join-total` | client-id | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | -| 18 | `consumer-coordinator-metrics` | `last-heartbeat-seconds-ago` | client-id | `kafka.consumer.last_heartbeat_seconds_ago` | The number of seconds since the last coordinator heartbeat was sent | `DOUBLE_OBSERVABLE_GAUGE` | -| 19 | `consumer-coordinator-metrics` | `last-rebalance-seconds-ago` | client-id | `kafka.consumer.last_rebalance_seconds_ago` | The number of seconds since the last successful rebalance event | `DOUBLE_OBSERVABLE_GAUGE` | -| 20 | `consumer-coordinator-metrics` | `partition-assigned-latency-avg` | client-id | `kafka.consumer.partition_assigned_latency_avg` | The average time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 21 | `consumer-coordinator-metrics` | `partition-assigned-latency-max` | client-id | `kafka.consumer.partition_assigned_latency_max` | The max time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 22 | `consumer-coordinator-metrics` | `partition-lost-latency-avg` | client-id | `kafka.consumer.partition_lost_latency_avg` | The average time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 23 | `consumer-coordinator-metrics` | `partition-lost-latency-max` | client-id | `kafka.consumer.partition_lost_latency_max` | The max time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 24 | `consumer-coordinator-metrics` | `partition-revoked-latency-avg` | client-id | `kafka.consumer.partition_revoked_latency_avg` | The average time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 25 | `consumer-coordinator-metrics` | `partition-revoked-latency-max` | client-id | `kafka.consumer.partition_revoked_latency_max` | The max time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 26 | `consumer-coordinator-metrics` | `rebalance-latency-avg` | client-id | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 27 | `consumer-coordinator-metrics` | `rebalance-latency-max` | client-id | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 28 | `consumer-coordinator-metrics` | `rebalance-latency-total` | client-id | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | -| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | client-id | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 30 | `consumer-coordinator-metrics` | `rebalance-total` | client-id | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | -| 31 | `consumer-coordinator-metrics` | `sync-rate` | client-id | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 32 | `consumer-coordinator-metrics` | `sync-time-avg` | client-id | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | -| 33 | `consumer-coordinator-metrics` | `sync-time-max` | client-id | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | -| 34 | `consumer-coordinator-metrics` | `sync-total` | client-id | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | -| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id,topic | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | client-id | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 37 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id,topic | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 38 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | client-id | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | client-id | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | client-id | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | client-id | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | client-id,topic | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | client-id,topic | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | client-id | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 46 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | client-id | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | client-id | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 48 | `consumer-fetch-manager-metrics` | `fetch-total` | client-id | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | -| 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | client-id,topic,partition | | | | -| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | client-id,topic | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | client-id,topic | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 54 | `consumer-fetch-manager-metrics` | `records-lag` | client-id,topic,partition | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 55 | `consumer-fetch-manager-metrics` | `records-lag-avg` | client-id,topic,partition | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | client-id | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | client-id,topic,partition | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 58 | `consumer-fetch-manager-metrics` | `records-lead` | client-id,topic,partition | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 59 | `consumer-fetch-manager-metrics` | `records-lead-avg` | client-id,topic,partition | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 60 | `consumer-fetch-manager-metrics` | `records-lead-min` | client-id | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 61 | `consumer-fetch-manager-metrics` | `records-lead-min` | client-id,topic,partition | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 62 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | -| 63 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | client-id,topic | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | -| 64 | `consumer-metrics` | `connection-close-rate` | client-id | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 65 | `consumer-metrics` | `connection-close-total` | client-id | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | -| 66 | `consumer-metrics` | `connection-count` | client-id | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 67 | `consumer-metrics` | `connection-creation-rate` | client-id | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 68 | `consumer-metrics` | `connection-creation-total` | client-id | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 69 | `consumer-metrics` | `failed-authentication-rate` | client-id | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 70 | `consumer-metrics` | `failed-authentication-total` | client-id | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 71 | `consumer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 72 | `consumer-metrics` | `failed-reauthentication-total` | client-id | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 73 | `consumer-metrics` | `incoming-byte-rate` | client-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 74 | `consumer-metrics` | `incoming-byte-total` | client-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 75 | `consumer-metrics` | `io-ratio` | client-id | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | -| 76 | `consumer-metrics` | `io-time-ns-avg` | client-id | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 77 | `consumer-metrics` | `io-wait-ratio` | client-id | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | -| 78 | `consumer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 79 | `consumer-metrics` | `io-waittime-total` | client-id | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | -| 80 | `consumer-metrics` | `iotime-total` | client-id | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | -| 81 | `consumer-metrics` | `last-poll-seconds-ago` | client-id | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | -| 82 | `consumer-metrics` | `network-io-rate` | client-id | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 83 | `consumer-metrics` | `network-io-total` | client-id | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 84 | `consumer-metrics` | `outgoing-byte-rate` | client-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 85 | `consumer-metrics` | `outgoing-byte-total` | client-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 86 | `consumer-metrics` | `poll-idle-ratio-avg` | client-id | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | -| 87 | `consumer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 88 | `consumer-metrics` | `reauthentication-latency-max` | client-id | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 89 | `consumer-metrics` | `request-rate` | client-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 90 | `consumer-metrics` | `request-size-avg` | client-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 91 | `consumer-metrics` | `request-size-max` | client-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 92 | `consumer-metrics` | `request-total` | client-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 93 | `consumer-metrics` | `response-rate` | client-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 94 | `consumer-metrics` | `response-total` | client-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 95 | `consumer-metrics` | `select-rate` | client-id | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 96 | `consumer-metrics` | `select-total` | client-id | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | -| 97 | `consumer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 98 | `consumer-metrics` | `successful-authentication-rate` | client-id | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 99 | `consumer-metrics` | `successful-authentication-total` | client-id | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 100 | `consumer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 101 | `consumer-metrics` | `successful-reauthentication-total` | client-id | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 102 | `consumer-metrics` | `time-between-poll-avg` | client-id | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | -| 103 | `consumer-metrics` | `time-between-poll-max` | client-id | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | -| 104 | `consumer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 105 | `consumer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 107 | `consumer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 108 | `consumer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | -| 109 | `consumer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | -| 110 | `consumer-node-metrics` | `request-rate` | client-id,node-id | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 111 | `consumer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 112 | `consumer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 113 | `consumer-node-metrics` | `request-total` | client-id,node-id | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 114 | `consumer-node-metrics` | `response-rate` | client-id,node-id | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 115 | `consumer-node-metrics` | `response-total` | client-id,node-id | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 116 | `kafka-metrics-count` | `count` | client-id | | | | -| 117 | `producer-metrics` | `batch-size-avg` | client-id | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 118 | `producer-metrics` | `batch-size-max` | client-id | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 119 | `producer-metrics` | `batch-split-rate` | client-id | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 120 | `producer-metrics` | `batch-split-total` | client-id | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | -| 121 | `producer-metrics` | `buffer-available-bytes` | client-id | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | -| 122 | `producer-metrics` | `buffer-exhausted-rate` | client-id | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_GAUGE` | -| 123 | `producer-metrics` | `buffer-exhausted-total` | client-id | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | -| 124 | `producer-metrics` | `buffer-total-bytes` | client-id | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | -| 125 | `producer-metrics` | `bufferpool-wait-ratio` | client-id | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_GAUGE` | -| 126 | `producer-metrics` | `bufferpool-wait-time-total` | client-id | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | -| 127 | `producer-metrics` | `compression-rate-avg` | client-id | `kafka.producer.compression_rate_avg` | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 128 | `producer-metrics` | `connection-close-rate` | client-id | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 129 | `producer-metrics` | `connection-close-total` | client-id | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | -| 130 | `producer-metrics` | `connection-count` | client-id | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 131 | `producer-metrics` | `connection-creation-rate` | client-id | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 132 | `producer-metrics` | `connection-creation-total` | client-id | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 133 | `producer-metrics` | `failed-authentication-rate` | client-id | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 134 | `producer-metrics` | `failed-authentication-total` | client-id | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 135 | `producer-metrics` | `failed-reauthentication-rate` | client-id | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 136 | `producer-metrics` | `failed-reauthentication-total` | client-id | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 137 | `producer-metrics` | `incoming-byte-rate` | client-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 138 | `producer-metrics` | `incoming-byte-total` | client-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 139 | `producer-metrics` | `io-ratio` | client-id | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | -| 140 | `producer-metrics` | `io-time-ns-avg` | client-id | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 141 | `producer-metrics` | `io-wait-ratio` | client-id | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | -| 142 | `producer-metrics` | `io-wait-time-ns-avg` | client-id | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 143 | `producer-metrics` | `io-waittime-total` | client-id | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | -| 144 | `producer-metrics` | `iotime-total` | client-id | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | -| 145 | `producer-metrics` | `metadata-age` | client-id | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | -| 146 | `producer-metrics` | `network-io-rate` | client-id | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 147 | `producer-metrics` | `network-io-total` | client-id | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 148 | `producer-metrics` | `outgoing-byte-rate` | client-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 149 | `producer-metrics` | `outgoing-byte-total` | client-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 150 | `producer-metrics` | `produce-throttle-time-avg` | client-id | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | -| 151 | `producer-metrics` | `produce-throttle-time-max` | client-id | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | -| 152 | `producer-metrics` | `reauthentication-latency-avg` | client-id | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 153 | `producer-metrics` | `reauthentication-latency-max` | client-id | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 154 | `producer-metrics` | `record-error-rate` | client-id | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | -| 155 | `producer-metrics` | `record-error-total` | client-id | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | -| 156 | `producer-metrics` | `record-queue-time-avg` | client-id | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | -| 157 | `producer-metrics` | `record-queue-time-max` | client-id | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | -| 158 | `producer-metrics` | `record-retry-rate` | client-id | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | -| 159 | `producer-metrics` | `record-retry-total` | client-id | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | -| 160 | `producer-metrics` | `record-send-rate` | client-id | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 161 | `producer-metrics` | `record-send-total` | client-id | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | -| 162 | `producer-metrics` | `record-size-avg` | client-id | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | -| 163 | `producer-metrics` | `record-size-max` | client-id | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | -| 164 | `producer-metrics` | `records-per-request-avg` | client-id | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 165 | `producer-metrics` | `request-latency-avg` | client-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 166 | `producer-metrics` | `request-latency-max` | client-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 167 | `producer-metrics` | `request-rate` | client-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 168 | `producer-metrics` | `request-size-avg` | client-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 169 | `producer-metrics` | `request-size-max` | client-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 170 | `producer-metrics` | `request-total` | client-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 171 | `producer-metrics` | `requests-in-flight` | client-id | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | -| 172 | `producer-metrics` | `response-rate` | client-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 173 | `producer-metrics` | `response-total` | client-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 174 | `producer-metrics` | `select-rate` | client-id | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 175 | `producer-metrics` | `select-total` | client-id | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | -| 176 | `producer-metrics` | `successful-authentication-no-reauth-total` | client-id | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 177 | `producer-metrics` | `successful-authentication-rate` | client-id | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 178 | `producer-metrics` | `successful-authentication-total` | client-id | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 179 | `producer-metrics` | `successful-reauthentication-rate` | client-id | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 180 | `producer-metrics` | `successful-reauthentication-total` | client-id | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 181 | `producer-metrics` | `waiting-threads` | client-id | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | -| 182 | `producer-node-metrics` | `incoming-byte-rate` | client-id,node-id | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 183 | `producer-node-metrics` | `incoming-byte-total` | client-id,node-id | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 184 | `producer-node-metrics` | `outgoing-byte-rate` | client-id,node-id | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 185 | `producer-node-metrics` | `outgoing-byte-total` | client-id,node-id | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 186 | `producer-node-metrics` | `request-latency-avg` | client-id,node-id | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 187 | `producer-node-metrics` | `request-latency-max` | client-id,node-id | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 188 | `producer-node-metrics` | `request-rate` | client-id,node-id | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 189 | `producer-node-metrics` | `request-size-avg` | client-id,node-id | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 190 | `producer-node-metrics` | `request-size-max` | client-id,node-id | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 191 | `producer-node-metrics` | `request-total` | client-id,node-id | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 192 | `producer-node-metrics` | `response-rate` | client-id,node-id | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 193 | `producer-node-metrics` | `response-total` | client-id,node-id | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 194 | `producer-topic-metrics` | `byte-rate` | client-id,topic | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | -| 195 | `producer-topic-metrics` | `byte-total` | client-id,topic | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | -| 196 | `producer-topic-metrics` | `compression-rate` | client-id,topic | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 197 | `producer-topic-metrics` | `record-error-rate` | client-id,topic | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | -| 198 | `producer-topic-metrics` | `record-error-total` | client-id,topic | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | -| 199 | `producer-topic-metrics` | `record-retry-rate` | client-id,topic | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | -| 200 | `producer-topic-metrics` | `record-retry-total` | client-id,topic | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | -| 201 | `producer-topic-metrics` | `record-send-rate` | client-id,topic | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 202 | `producer-topic-metrics` | `record-send-total` | client-id,topic | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | +| 1 | `app-info` | `commit-id` | `client-id` | | | | +| 2 | `app-info` | `start-time-ms` | `client-id` | | | | +| 3 | `app-info` | `version` | `client-id` | | | | +| 4 | `consumer-coordinator-metrics` | `assigned-partitions` | `client-id` | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | +| 5 | `consumer-coordinator-metrics` | `commit-latency-avg` | `client-id` | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| 6 | `consumer-coordinator-metrics` | `commit-latency-max` | `client-id` | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| 7 | `consumer-coordinator-metrics` | `commit-rate` | `client-id` | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 8 | `consumer-coordinator-metrics` | `commit-total` | `client-id` | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | +| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | `client-id` | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_GAUGE` | +| 10 | `consumer-coordinator-metrics` | `failed-rebalance-total` | `client-id` | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | +| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | `client-id` | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 12 | `consumer-coordinator-metrics` | `heartbeat-response-time-max` | `client-id` | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | +| 13 | `consumer-coordinator-metrics` | `heartbeat-total` | `client-id` | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | +| 14 | `consumer-coordinator-metrics` | `join-rate` | `client-id` | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 15 | `consumer-coordinator-metrics` | `join-time-avg` | `client-id` | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| 16 | `consumer-coordinator-metrics` | `join-time-max` | `client-id` | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| 17 | `consumer-coordinator-metrics` | `join-total` | `client-id` | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | +| 18 | `consumer-coordinator-metrics` | `last-heartbeat-seconds-ago` | `client-id` | `kafka.consumer.last_heartbeat_seconds_ago` | The number of seconds since the last coordinator heartbeat was sent | `DOUBLE_OBSERVABLE_GAUGE` | +| 19 | `consumer-coordinator-metrics` | `last-rebalance-seconds-ago` | `client-id` | `kafka.consumer.last_rebalance_seconds_ago` | The number of seconds since the last successful rebalance event | `DOUBLE_OBSERVABLE_GAUGE` | +| 20 | `consumer-coordinator-metrics` | `partition-assigned-latency-avg` | `client-id` | `kafka.consumer.partition_assigned_latency_avg` | The average time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 21 | `consumer-coordinator-metrics` | `partition-assigned-latency-max` | `client-id` | `kafka.consumer.partition_assigned_latency_max` | The max time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 22 | `consumer-coordinator-metrics` | `partition-lost-latency-avg` | `client-id` | `kafka.consumer.partition_lost_latency_avg` | The average time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 23 | `consumer-coordinator-metrics` | `partition-lost-latency-max` | `client-id` | `kafka.consumer.partition_lost_latency_max` | The max time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 24 | `consumer-coordinator-metrics` | `partition-revoked-latency-avg` | `client-id` | `kafka.consumer.partition_revoked_latency_avg` | The average time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 25 | `consumer-coordinator-metrics` | `partition-revoked-latency-max` | `client-id` | `kafka.consumer.partition_revoked_latency_max` | The max time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| 26 | `consumer-coordinator-metrics` | `rebalance-latency-avg` | `client-id` | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| 27 | `consumer-coordinator-metrics` | `rebalance-latency-max` | `client-id` | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| 28 | `consumer-coordinator-metrics` | `rebalance-latency-total` | `client-id` | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | +| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | `client-id` | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| 30 | `consumer-coordinator-metrics` | `rebalance-total` | `client-id` | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | +| 31 | `consumer-coordinator-metrics` | `sync-rate` | `client-id` | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 32 | `consumer-coordinator-metrics` | `sync-time-avg` | `client-id` | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| 33 | `consumer-coordinator-metrics` | `sync-time-max` | `client-id` | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| 34 | `consumer-coordinator-metrics` | `sync-total` | `client-id` | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | +| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id` | | | | +| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 37 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id` | | | | +| 38 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | `client-id` | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | `client-id` | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | `client-id` | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | +| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id`,`topic` | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id` | | | | +| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id`,`topic` | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id` | | | | +| 46 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | `client-id` | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | `client-id` | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 48 | `consumer-fetch-manager-metrics` | `fetch-total` | `client-id` | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | +| 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | `client-id`,`topic`,`partition` | | | | +| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id`,`topic` | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id` | | | | +| 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id` | | | | +| 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id`,`topic` | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| 54 | `consumer-fetch-manager-metrics` | `records-lag` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 55 | `consumer-fetch-manager-metrics` | `records-lag-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id` | | | | +| 58 | `consumer-fetch-manager-metrics` | `records-lead` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 59 | `consumer-fetch-manager-metrics` | `records-lead-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| 60 | `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id` | | | | +| 61 | `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| 62 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id` | | | | +| 63 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id`,`topic` | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | +| 64 | `consumer-metrics` | `connection-close-rate` | `client-id` | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 65 | `consumer-metrics` | `connection-close-total` | `client-id` | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| 66 | `consumer-metrics` | `connection-count` | `client-id` | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| 67 | `consumer-metrics` | `connection-creation-rate` | `client-id` | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 68 | `consumer-metrics` | `connection-creation-total` | `client-id` | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| 69 | `consumer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 70 | `consumer-metrics` | `failed-authentication-total` | `client-id` | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 71 | `consumer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 72 | `consumer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 73 | `consumer-metrics` | `incoming-byte-rate` | `client-id` | | | | +| 74 | `consumer-metrics` | `incoming-byte-total` | `client-id` | | | | +| 75 | `consumer-metrics` | `io-ratio` | `client-id` | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | +| 76 | `consumer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 77 | `consumer-metrics` | `io-wait-ratio` | `client-id` | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | +| 78 | `consumer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 79 | `consumer-metrics` | `io-waittime-total` | `client-id` | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| 80 | `consumer-metrics` | `iotime-total` | `client-id` | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| 81 | `consumer-metrics` | `last-poll-seconds-ago` | `client-id` | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | +| 82 | `consumer-metrics` | `network-io-rate` | `client-id` | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 83 | `consumer-metrics` | `network-io-total` | `client-id` | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 84 | `consumer-metrics` | `outgoing-byte-rate` | `client-id` | | | | +| 85 | `consumer-metrics` | `outgoing-byte-total` | `client-id` | | | | +| 86 | `consumer-metrics` | `poll-idle-ratio-avg` | `client-id` | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | +| 87 | `consumer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 88 | `consumer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 89 | `consumer-metrics` | `request-rate` | `client-id` | | | | +| 90 | `consumer-metrics` | `request-size-avg` | `client-id` | | | | +| 91 | `consumer-metrics` | `request-size-max` | `client-id` | | | | +| 92 | `consumer-metrics` | `request-total` | `client-id` | | | | +| 93 | `consumer-metrics` | `response-rate` | `client-id` | | | | +| 94 | `consumer-metrics` | `response-total` | `client-id` | | | | +| 95 | `consumer-metrics` | `select-rate` | `client-id` | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 96 | `consumer-metrics` | `select-total` | `client-id` | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| 97 | `consumer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 98 | `consumer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 99 | `consumer-metrics` | `successful-authentication-total` | `client-id` | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 100 | `consumer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 101 | `consumer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 102 | `consumer-metrics` | `time-between-poll-avg` | `client-id` | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| 103 | `consumer-metrics` | `time-between-poll-max` | `client-id` | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| 104 | `consumer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 105 | `consumer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 107 | `consumer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 108 | `consumer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | +| 109 | `consumer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | +| 110 | `consumer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 111 | `consumer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 112 | `consumer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 113 | `consumer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 114 | `consumer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 115 | `consumer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 116 | `kafka-metrics-count` | `count` | `client-id` | | | | +| 117 | `producer-metrics` | `batch-size-avg` | `client-id` | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 118 | `producer-metrics` | `batch-size-max` | `client-id` | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 119 | `producer-metrics` | `batch-split-rate` | `client-id` | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 120 | `producer-metrics` | `batch-split-total` | `client-id` | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | +| 121 | `producer-metrics` | `buffer-available-bytes` | `client-id` | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | +| 122 | `producer-metrics` | `buffer-exhausted-rate` | `client-id` | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_GAUGE` | +| 123 | `producer-metrics` | `buffer-exhausted-total` | `client-id` | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | +| 124 | `producer-metrics` | `buffer-total-bytes` | `client-id` | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | +| 125 | `producer-metrics` | `bufferpool-wait-ratio` | `client-id` | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_GAUGE` | +| 126 | `producer-metrics` | `bufferpool-wait-time-total` | `client-id` | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | +| 127 | `producer-metrics` | `compression-rate-avg` | `client-id` | `kafka.producer.compression_rate_avg` | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | +| 128 | `producer-metrics` | `connection-close-rate` | `client-id` | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 129 | `producer-metrics` | `connection-close-total` | `client-id` | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| 130 | `producer-metrics` | `connection-count` | `client-id` | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| 131 | `producer-metrics` | `connection-creation-rate` | `client-id` | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 132 | `producer-metrics` | `connection-creation-total` | `client-id` | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| 133 | `producer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 134 | `producer-metrics` | `failed-authentication-total` | `client-id` | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 135 | `producer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 136 | `producer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 137 | `producer-metrics` | `incoming-byte-rate` | `client-id` | | | | +| 138 | `producer-metrics` | `incoming-byte-total` | `client-id` | | | | +| 139 | `producer-metrics` | `io-ratio` | `client-id` | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | +| 140 | `producer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 141 | `producer-metrics` | `io-wait-ratio` | `client-id` | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | +| 142 | `producer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| 143 | `producer-metrics` | `io-waittime-total` | `client-id` | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| 144 | `producer-metrics` | `iotime-total` | `client-id` | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| 145 | `producer-metrics` | `metadata-age` | `client-id` | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | +| 146 | `producer-metrics` | `network-io-rate` | `client-id` | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 147 | `producer-metrics` | `network-io-total` | `client-id` | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 148 | `producer-metrics` | `outgoing-byte-rate` | `client-id` | | | | +| 149 | `producer-metrics` | `outgoing-byte-total` | `client-id` | | | | +| 150 | `producer-metrics` | `produce-throttle-time-avg` | `client-id` | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| 151 | `producer-metrics` | `produce-throttle-time-max` | `client-id` | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| 152 | `producer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 153 | `producer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| 154 | `producer-metrics` | `record-error-rate` | `client-id` | | | | +| 155 | `producer-metrics` | `record-error-total` | `client-id` | | | | +| 156 | `producer-metrics` | `record-queue-time-avg` | `client-id` | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| 157 | `producer-metrics` | `record-queue-time-max` | `client-id` | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| 158 | `producer-metrics` | `record-retry-rate` | `client-id` | | | | +| 159 | `producer-metrics` | `record-retry-total` | `client-id` | | | | +| 160 | `producer-metrics` | `record-send-rate` | `client-id` | | | | +| 161 | `producer-metrics` | `record-send-total` | `client-id` | | | | +| 162 | `producer-metrics` | `record-size-avg` | `client-id` | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | +| 163 | `producer-metrics` | `record-size-max` | `client-id` | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | +| 164 | `producer-metrics` | `records-per-request-avg` | `client-id` | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | +| 165 | `producer-metrics` | `request-latency-avg` | `client-id` | | | | +| 166 | `producer-metrics` | `request-latency-max` | `client-id` | | | | +| 167 | `producer-metrics` | `request-rate` | `client-id` | | | | +| 168 | `producer-metrics` | `request-size-avg` | `client-id` | | | | +| 169 | `producer-metrics` | `request-size-max` | `client-id` | | | | +| 170 | `producer-metrics` | `request-total` | `client-id` | | | | +| 171 | `producer-metrics` | `requests-in-flight` | `client-id` | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | +| 172 | `producer-metrics` | `response-rate` | `client-id` | | | | +| 173 | `producer-metrics` | `response-total` | `client-id` | | | | +| 174 | `producer-metrics` | `select-rate` | `client-id` | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 175 | `producer-metrics` | `select-total` | `client-id` | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| 176 | `producer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 177 | `producer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 178 | `producer-metrics` | `successful-authentication-total` | `client-id` | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| 179 | `producer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 180 | `producer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| 181 | `producer-metrics` | `waiting-threads` | `client-id` | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | +| 182 | `producer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 183 | `producer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| 184 | `producer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 185 | `producer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| 186 | `producer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 187 | `producer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| 188 | `producer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 189 | `producer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 190 | `producer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| 191 | `producer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| 192 | `producer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | +| 193 | `producer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| 194 | `producer-topic-metrics` | `byte-rate` | `client-id`,`topic` | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | +| 195 | `producer-topic-metrics` | `byte-total` | `client-id`,`topic` | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | +| 196 | `producer-topic-metrics` | `compression-rate` | `client-id`,`topic` | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | +| 197 | `producer-topic-metrics` | `record-error-rate` | `client-id`,`topic` | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | +| 198 | `producer-topic-metrics` | `record-error-total` | `client-id`,`topic` | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | +| 199 | `producer-topic-metrics` | `record-retry-rate` | `client-id`,`topic` | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | +| 200 | `producer-topic-metrics` | `record-retry-total` | `client-id`,`topic` | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | +| 201 | `producer-topic-metrics` | `record-send-rate` | `client-id`,`topic` | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | +| 202 | `producer-topic-metrics` | `record-send-total` | `client-id`,`topic` | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index c0c052a82768..50874653d7b7 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -8,6 +8,10 @@ import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; +import io.opentelemetry.api.common.Attributes; +import io.opentelemetry.api.common.AttributesBuilder; +import io.opentelemetry.api.metrics.Meter; +import io.opentelemetry.api.metrics.ObservableDoubleMeasurement; import java.util.Arrays; import java.util.HashMap; import java.util.HashSet; @@ -15,7 +19,9 @@ import java.util.Optional; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; import javax.annotation.Nullable; +import org.apache.kafka.common.metrics.KafkaMetric; /** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */ class KafkaMetricRegistry { @@ -51,7 +57,8 @@ class KafkaMetricRegistry { } @Nullable - static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) { + static RegisteredObservable getRegisteredObservable( + Meter meter, KafkaMetricId kafkaMetricId, KafkaMetric kafkaMetric) { // If metric is not a Measureable, we can't map it to an instrument if (kafkaMetricId.getMeasureable() == null) { return null; @@ -64,19 +71,61 @@ static InstrumentDescriptor getInstrumentDescriptor(KafkaMetricId kafkaMetricId) } String instrumentName = "kafka." + matchingGroup.get() + "." + kafkaMetricId.getName().replace("-", "_"); - String description = + String instrumentDescription = descriptionCache.computeIfAbsent(instrumentName, s -> kafkaMetricId.getDescription()); String instrumentType = measureableToInstrumentType.getOrDefault( kafkaMetricId.getMeasureable(), INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + InstrumentDescriptor instrumentDescriptor = + toInstrumentDescriptor(instrumentType, instrumentName, instrumentDescription); + Attributes attributes = toAttributes(kafkaMetric); + AutoCloseable observable = + createObservable(meter, attributes, instrumentDescriptor, kafkaMetric); + return RegisteredObservable.create(kafkaMetricId, instrumentDescriptor, attributes, observable); + } + + private static InstrumentDescriptor toInstrumentDescriptor( + String instrumentType, String instrumentName, String instrumentDescription) { switch (instrumentType) { case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: - return InstrumentDescriptor.createDoubleGauge(instrumentName, description); + return InstrumentDescriptor.createDoubleGauge(instrumentName, instrumentDescription); + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: + return InstrumentDescriptor.createDoubleCounter(instrumentName, instrumentDescription); + default: // Continue below to throw + } + throw new IllegalStateException("Unrecognized instrument type. This is a bug."); + } + + static Attributes toAttributes(KafkaMetric kafkaMetric) { + AttributesBuilder attributesBuilder = Attributes.builder(); + kafkaMetric.metricName().tags().forEach(attributesBuilder::put); + return attributesBuilder.build(); + } + + private static AutoCloseable createObservable( + Meter meter, + Attributes attributes, + InstrumentDescriptor instrumentDescriptor, + KafkaMetric kafkaMetric) { + Consumer callback = + observableMeasurement -> observableMeasurement.record(kafkaMetric.value(), attributes); + switch (instrumentDescriptor.getInstrumentType()) { + case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: + return meter + .gaugeBuilder(instrumentDescriptor.getName()) + .setDescription(instrumentDescriptor.getDescription()) + .buildWithCallback(callback); case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: - return InstrumentDescriptor.createDoubleCounter(instrumentName, description); + return meter + .counterBuilder(instrumentDescriptor.getName()) + .setDescription(instrumentDescriptor.getDescription()) + .ofDoubles() + .buildWithCallback(callback); default: // Continue below to throw } + // TODO: add support for other instrument types and value types as needed for new instruments. + // This should not happen. throw new IllegalStateException("Unrecognized instrument type. This is a bug."); } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 1174eb26b905..8ad093fa3103 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -5,26 +5,15 @@ package io.opentelemetry.instrumentation.kafkaclients; -import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER; -import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE; -import static java.lang.System.lineSeparator; -import static java.util.Comparator.comparing; -import static java.util.stream.Collectors.groupingBy; -import static java.util.stream.Collectors.joining; -import static java.util.stream.Collectors.toList; - import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.common.AttributeKey; import io.opentelemetry.api.common.Attributes; -import io.opentelemetry.api.common.AttributesBuilder; import io.opentelemetry.api.metrics.Meter; -import io.opentelemetry.api.metrics.ObservableDoubleMeasurement; +import java.util.ArrayList; import java.util.Iterator; import java.util.List; import java.util.Map; -import java.util.Optional; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.function.Consumer; import java.util.logging.Level; import java.util.logging.Logger; import javax.annotation.Nullable; @@ -49,12 +38,9 @@ public class OpenTelemetryKafkaMetrics implements MetricsReporter { private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName()); - private static final Set seenMetrics = ConcurrentHashMap.newKeySet(); - @Nullable private static volatile Meter meter; - private static final Map instrumentMap = - new ConcurrentHashMap<>(); + private static final List registeredObservables = new ArrayList<>(); /** * Setup OpenTelemetry. This should be called as early in the application lifecycle as possible. @@ -73,48 +59,9 @@ static void resetForTest() { closeAllInstruments(); } - /** - * Print a table mapping kafka metrics to equivalent OpenTelemetry metrics, in markdown format. - */ - static void printMappingTable() { - StringBuilder sb = new StringBuilder(); - // Append table headers - sb.append( - "| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type |") - .append(lineSeparator()) - .append( - "|---|--------------|-------------|----------------|-----------------|------------------------|-----------------|") - .append(lineSeparator()); - Map> kafkaMetricsByGroup = - seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup)); - int count = 1; - // Iterate through groups in alpha order - for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { - List kafkaMetricIds = - kafkaMetricsByGroup.get(group).stream() - .sorted(comparing(KafkaMetricId::getName)) - .collect(toList()); - // Iterate through metrics in alpha order by name - for (KafkaMetricId kafkaMetricId : kafkaMetricIds) { - Optional descriptor = - Optional.ofNullable(KafkaMetricRegistry.getInstrumentDescriptor(kafkaMetricId)); - // Append table row - sb.append( - String.format( - "| %s | %s | %s | %s | %s | %s | %s |%n", - count, - "`" + group + "`", - "`" + kafkaMetricId.getName() + "`", - kafkaMetricId.getAttributeKeys().stream() - .map(key -> "`" + key + "`") - .collect(joining(",")), - descriptor.map(i -> "`" + i.getName() + "`").orElse(""), - descriptor.map(InstrumentDescriptor::getDescription).orElse(""), - descriptor.map(i -> "`" + i.getInstrumentType() + "`").orElse(""))); - count++; - } - } - logger.log(Level.INFO, "Mapping table" + System.lineSeparator() + sb); + // Visible for test + static List getRegisteredObservables() { + return registeredObservables; } @Override @@ -123,86 +70,58 @@ public void init(List metrics) { } @Override - public void metricChange(KafkaMetric metric) { + public synchronized void metricChange(KafkaMetric metric) { KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); - seenMetrics.add(kafkaMetricId); Meter currentMeter = meter; if (currentMeter == null) { logger.log(Level.FINEST, "Metric changed but meter not set: {0}", kafkaMetricId); return; } - InstrumentDescriptor instrumentDescriptor = - KafkaMetricRegistry.getInstrumentDescriptor(kafkaMetricId); - if (instrumentDescriptor == null) { + RegisteredObservable registeredObservable = + KafkaMetricRegistry.getRegisteredObservable(currentMeter, kafkaMetricId, metric); + if (registeredObservable == null) { logger.log(Level.FINEST, "Metric changed but cannot map to instrument: {0}", kafkaMetricId); return; } - RegisteredInstrument registeredInstrument = - RegisteredInstrument.create(kafkaMetricId, toAttributes(metric)); - - instrumentMap.compute( - registeredInstrument, - (registeredInstrument1, autoCloseable) -> { - if (autoCloseable != null) { - logger.log(Level.FINEST, "Replacing instrument {0}", registeredInstrument1); - try { - autoCloseable.close(); - } catch (Exception e) { - logger.log(Level.WARNING, "An error occurred closing instrument", e); - } - } else { - logger.log(Level.FINEST, "Adding instrument {0}", registeredInstrument1); - } - return createObservable( - currentMeter, registeredInstrument1, instrumentDescriptor, metric); - }); - } - - private static AutoCloseable createObservable( - Meter meter, - RegisteredInstrument registeredInstrument, - InstrumentDescriptor instrumentDescriptor, - KafkaMetric kafkaMetric) { - Consumer callback = - observableMeasurement -> - observableMeasurement.record(kafkaMetric.value(), registeredInstrument.getAttributes()); - switch (instrumentDescriptor.getInstrumentType()) { - case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE: - return meter - .gaugeBuilder(instrumentDescriptor.getName()) - .setDescription(instrumentDescriptor.getDescription()) - .buildWithCallback(callback); - case INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER: - return meter - .counterBuilder(instrumentDescriptor.getName()) - .setDescription(instrumentDescriptor.getDescription()) - .ofDoubles() - .buildWithCallback(callback); - default: // Continue below to throw + InstrumentDescriptor instrumentDescriptor = registeredObservable.getInstrumentDescriptor(); + Attributes attributes = registeredObservable.getAttributes(); + Set> attributeKeys = attributes.asMap().keySet(); + + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + RegisteredObservable curRegisteredObservable = it.next(); + Set> curAttributeKeys = + curRegisteredObservable.getAttributes().asMap().keySet(); + if (curRegisteredObservable.equals(registeredObservable)) { + logger.log(Level.FINEST, "Replacing instrument: {0}", curRegisteredObservable); + closeInstrument(curRegisteredObservable.getObservable()); + it.remove(); + } else if (curRegisteredObservable.getInstrumentDescriptor().equals(instrumentDescriptor) + && attributeKeys.size() > curAttributeKeys.size() + && attributeKeys.containsAll(curAttributeKeys)) { + logger.log( + Level.FINEST, + "Replacing instrument with higher dimension version: {0}", + curRegisteredObservable); + closeInstrument(curRegisteredObservable.getObservable()); + it.remove(); + } } - // TODO: add support for other instrument types and value types as needed for new instruments - // registered in KafkaMetricRegistry. - // This should not happen. - throw new IllegalStateException("Unrecognized instrument type. This is a bug."); - } - private static Attributes toAttributes(KafkaMetric kafkaMetric) { - AttributesBuilder attributesBuilder = Attributes.builder(); - kafkaMetric.metricName().tags().forEach(attributesBuilder::put); - return attributesBuilder.build(); + registeredObservables.add(registeredObservable); } @Override public void metricRemoval(KafkaMetric metric) { KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); - seenMetrics.add(kafkaMetricId); logger.log(Level.FINEST, "Metric removed: {0}", kafkaMetricId); - AutoCloseable observable = - instrumentMap.remove(RegisteredInstrument.create(kafkaMetricId, toAttributes(metric))); - if (observable != null) { - closeInstrument(observable); + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + RegisteredObservable current = it.next(); + if (current.matches(metric)) { + closeInstrument(current.getObservable()); + it.remove(); + } } } @@ -212,10 +131,8 @@ public void close() { } static void closeAllInstruments() { - for (Iterator> it = - instrumentMap.entrySet().iterator(); - it.hasNext(); ) { - closeInstrument(it.next().getValue()); + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + closeInstrument(it.next().getObservable()); it.remove(); } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java deleted file mode 100644 index 341ed5bba7eb..000000000000 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredInstrument.java +++ /dev/null @@ -1,21 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.instrumentation.kafkaclients; - -import com.google.auto.value.AutoValue; -import io.opentelemetry.api.common.Attributes; - -@AutoValue -abstract class RegisteredInstrument { - - abstract KafkaMetricId getKafkaMetricId(); - - abstract Attributes getAttributes(); - - static RegisteredInstrument create(KafkaMetricId kafkaMetricId, Attributes attributes) { - return new AutoValue_RegisteredInstrument(kafkaMetricId, attributes); - } -} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java new file mode 100644 index 000000000000..824d7c2cbc01 --- /dev/null +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java @@ -0,0 +1,38 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.instrumentation.kafkaclients; + +import static io.opentelemetry.instrumentation.kafkaclients.KafkaMetricRegistry.toAttributes; + +import com.google.auto.value.AutoValue; +import io.opentelemetry.api.common.Attributes; +import org.apache.kafka.common.metrics.KafkaMetric; + +@AutoValue +abstract class RegisteredObservable { + + abstract KafkaMetricId getKafkaMetricId(); + + abstract InstrumentDescriptor getInstrumentDescriptor(); + + abstract Attributes getAttributes(); + + abstract AutoCloseable getObservable(); + + boolean matches(KafkaMetric kafkaMetric) { + return getKafkaMetricId().equals(KafkaMetricId.create(kafkaMetric)) + && getAttributes().equals(toAttributes(kafkaMetric)); + } + + static RegisteredObservable create( + KafkaMetricId kafkaMetricId, + InstrumentDescriptor instrumentDescriptor, + Attributes attributes, + AutoCloseable observable) { + return new AutoValue_RegisteredObservable( + kafkaMetricId, instrumentDescriptor, attributes, observable); + } +} From 9a4758e14114a8125bb0aabdbec23ca837f58c61 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Thu, 23 Jun 2022 12:35:13 -0500 Subject: [PATCH 10/17] PR feedback --- .../testing/build.gradle.kts | 3 + .../OpenTelemetryKafkaMetricsTest.java | 40 ++++++--- .../kafka-clients-common/library/README.md | 16 ++-- .../kafkaclients/KafkaMetricId.java | 43 --------- .../kafkaclients/KafkaMetricRegistry.java | 34 +++++--- .../OpenTelemetryKafkaMetrics.java | 87 ++++++++++--------- .../kafkaclients/RegisteredObservable.java | 15 +--- 7 files changed, 115 insertions(+), 123 deletions(-) delete mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts index d48496577f4b..1bd73daac2aa 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts @@ -12,5 +12,8 @@ dependencies { implementation("org.testcontainers:kafka") implementation("org.testcontainers:junit-jupiter") + compileOnly("com.google.auto.value:auto-value-annotations") + annotationProcessor("com.google.auto.value:auto-value") + runtimeOnly("org.apache.kafka:kafka_2.13:2.8.1") } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index a962763041c1..c5c640dcfc68 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -13,6 +13,7 @@ import static java.util.stream.Collectors.toSet; import static org.assertj.core.api.Assertions.assertThat; +import com.google.auto.value.AutoValue; import io.opentelemetry.api.common.AttributeKey; import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension; import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension; @@ -27,16 +28,15 @@ import java.util.HashSet; import java.util.List; import java.util.Map; -import java.util.Objects; import java.util.Optional; import java.util.Random; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.MetricName; import org.apache.kafka.common.metrics.KafkaMetric; import org.apache.kafka.common.metrics.MetricsReporter; import org.apache.kafka.common.serialization.ByteArrayDeserializer; @@ -373,8 +373,7 @@ private static void printMappingTable() { "|---|--------------|-------------|----------------|-----------------|------------------------|-----------------|") .append(lineSeparator()); Map> kafkaMetricsByGroup = - TestMetricsReporter.seenMetrics.keySet().stream() - .collect(groupingBy(KafkaMetricId::getGroup)); + TestMetricsReporter.seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup)); List registeredObservables = OpenTelemetryKafkaMetrics.getRegisteredObservables(); int count = 1; @@ -382,16 +381,20 @@ private static void printMappingTable() { for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { List kafkaMetricIds = kafkaMetricsByGroup.get(group).stream() - .sorted(comparing(KafkaMetricId::getName)) + .sorted( + comparing(KafkaMetricId::getName) + .thenComparing(kafkaMetricId -> kafkaMetricId.getAttributeKeys().size())) .collect(toList()); // Iterate through metrics in alpha order by name for (KafkaMetricId kafkaMetricId : kafkaMetricIds) { - KafkaMetric kafkaMetric = - Objects.requireNonNull(TestMetricsReporter.seenMetrics.get(kafkaMetricId)); + // Find first (there may be multiple) registered instrument that matches the kafkaMetricId Optional descriptor = registeredObservables.stream() - .filter(registeredObservable -> registeredObservable.matches(kafkaMetric)) - .findAny() + .filter( + registeredObservable -> + KafkaMetricId.create(registeredObservable.getKafkaMetricName()) + .equals(kafkaMetricId)) + .findFirst() .map(RegisteredObservable::getInstrumentDescriptor); // Append table row sb.append( @@ -414,7 +417,7 @@ private static void printMappingTable() { public static class TestMetricsReporter implements MetricsReporter { - private static final Map seenMetrics = new ConcurrentHashMap<>(); + private static final Set seenMetrics = new HashSet<>(); @Override public void init(List list) { @@ -423,7 +426,7 @@ public void init(List list) { @Override public void metricChange(KafkaMetric kafkaMetric) { - seenMetrics.put(KafkaMetricId.create(kafkaMetric), kafkaMetric); + seenMetrics.add(KafkaMetricId.create(kafkaMetric.metricName())); } @Override @@ -435,4 +438,19 @@ public void close() {} @Override public void configure(Map map) {} } + + @AutoValue + abstract static class KafkaMetricId { + + abstract String getGroup(); + + abstract String getName(); + + abstract Set getAttributeKeys(); + + static KafkaMetricId create(MetricName metricName) { + return new AutoValue_OpenTelemetryKafkaMetricsTest_KafkaMetricId( + metricName.group(), metricName.name(), metricName.tags().keySet()); + } + } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md index ad2d004f8f43..eb8502d00777 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md @@ -70,22 +70,22 @@ collected. | 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | `client-id` | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | | 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | `client-id` | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | | 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | `client-id` | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id`,`topic` | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id` | | | | -| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id`,`topic` | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id` | | | | +| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id` | | | | +| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id`,`topic` | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id` | | | | +| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id`,`topic` | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | | 46 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | `client-id` | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | | 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | `client-id` | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | | 48 | `consumer-fetch-manager-metrics` | `fetch-total` | `client-id` | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | | 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | `client-id`,`topic`,`partition` | | | | -| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id`,`topic` | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id` | | | | +| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id` | | | | +| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id`,`topic` | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | | 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id` | | | | | 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id`,`topic` | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | | 54 | `consumer-fetch-manager-metrics` | `records-lag` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | | 55 | `consumer-fetch-manager-metrics` | `records-lag-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id` | | | | +| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id` | | | | +| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | | 58 | `consumer-fetch-manager-metrics` | `records-lead` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | | 59 | `consumer-fetch-manager-metrics` | `records-lead-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | | 60 | `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id` | | | | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java deleted file mode 100644 index 71c5166abcd9..000000000000 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricId.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.instrumentation.kafkaclients; - -import com.google.auto.value.AutoValue; -import java.util.Set; -import javax.annotation.Nullable; -import org.apache.kafka.common.metrics.KafkaMetric; -import org.apache.kafka.common.metrics.Measurable; - -/** A value class collecting the identifying fields of a kafka {@link KafkaMetric}. */ -@AutoValue -abstract class KafkaMetricId { - - abstract String getGroup(); - - abstract String getName(); - - abstract String getDescription(); - - @Nullable - abstract Class getMeasureable(); - - abstract Set getAttributeKeys(); - - static KafkaMetricId create(KafkaMetric kafkaMetric) { - Class measureable; - try { - measureable = kafkaMetric.measurable().getClass(); - } catch (IllegalStateException e) { - measureable = null; - } - return new AutoValue_KafkaMetricId( - kafkaMetric.metricName().group(), - kafkaMetric.metricName().name(), - kafkaMetric.metricName().description(), - measureable, - kafkaMetric.metricName().tags().keySet()); - } -} diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java index 50874653d7b7..97ac761e171c 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java @@ -21,7 +21,9 @@ import java.util.concurrent.ConcurrentHashMap; import java.util.function.Consumer; import javax.annotation.Nullable; +import org.apache.kafka.common.MetricName; import org.apache.kafka.common.metrics.KafkaMetric; +import org.apache.kafka.common.metrics.Measurable; /** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */ class KafkaMetricRegistry { @@ -57,32 +59,42 @@ class KafkaMetricRegistry { } @Nullable - static RegisteredObservable getRegisteredObservable( - Meter meter, KafkaMetricId kafkaMetricId, KafkaMetric kafkaMetric) { + static RegisteredObservable getRegisteredObservable(Meter meter, KafkaMetric kafkaMetric) { // If metric is not a Measureable, we can't map it to an instrument - if (kafkaMetricId.getMeasureable() == null) { + Class measurable = getMeasurable(kafkaMetric); + if (measurable == null) { return null; } + MetricName metricName = kafkaMetric.metricName(); Optional matchingGroup = - groups.stream().filter(group -> kafkaMetricId.getGroup().contains(group)).findFirst(); + groups.stream().filter(group -> metricName.group().contains(group)).findFirst(); // Only map metrics that have a matching group if (!matchingGroup.isPresent()) { return null; } String instrumentName = - "kafka." + matchingGroup.get() + "." + kafkaMetricId.getName().replace("-", "_"); + "kafka." + matchingGroup.get() + "." + metricName.name().replace("-", "_"); String instrumentDescription = - descriptionCache.computeIfAbsent(instrumentName, s -> kafkaMetricId.getDescription()); + descriptionCache.computeIfAbsent(instrumentName, s -> metricName.description()); String instrumentType = measureableToInstrumentType.getOrDefault( - kafkaMetricId.getMeasureable(), INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); + measurable, INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE); InstrumentDescriptor instrumentDescriptor = toInstrumentDescriptor(instrumentType, instrumentName, instrumentDescription); - Attributes attributes = toAttributes(kafkaMetric); + Attributes attributes = toAttributes(metricName.tags()); AutoCloseable observable = createObservable(meter, attributes, instrumentDescriptor, kafkaMetric); - return RegisteredObservable.create(kafkaMetricId, instrumentDescriptor, attributes, observable); + return RegisteredObservable.create(metricName, instrumentDescriptor, attributes, observable); + } + + @Nullable + private static Class getMeasurable(KafkaMetric kafkaMetric) { + try { + return kafkaMetric.measurable().getClass(); + } catch (IllegalStateException e) { + return null; + } } private static InstrumentDescriptor toInstrumentDescriptor( @@ -97,9 +109,9 @@ private static InstrumentDescriptor toInstrumentDescriptor( throw new IllegalStateException("Unrecognized instrument type. This is a bug."); } - static Attributes toAttributes(KafkaMetric kafkaMetric) { + private static Attributes toAttributes(Map tags) { AttributesBuilder attributesBuilder = Attributes.builder(); - kafkaMetric.metricName().tags().forEach(attributesBuilder::put); + tags.forEach(attributesBuilder::put); return attributesBuilder.build(); } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 8ad093fa3103..a289071879d1 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -7,8 +7,8 @@ import io.opentelemetry.api.OpenTelemetry; import io.opentelemetry.api.common.AttributeKey; -import io.opentelemetry.api.common.Attributes; import io.opentelemetry.api.metrics.Meter; +import io.opentelemetry.instrumentation.api.internal.GuardedBy; import java.util.ArrayList; import java.util.Iterator; import java.util.List; @@ -40,6 +40,9 @@ public class OpenTelemetryKafkaMetrics implements MetricsReporter { private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName()); @Nullable private static volatile Meter meter; + private static final Object lock = new Object(); + + @GuardedBy("lock") private static final List registeredObservables = new ArrayList<>(); /** @@ -61,7 +64,9 @@ static void resetForTest() { // Visible for test static List getRegisteredObservables() { - return registeredObservables; + synchronized (lock) { + return new ArrayList<>(registeredObservables); + } } @Override @@ -71,56 +76,58 @@ public void init(List metrics) { @Override public synchronized void metricChange(KafkaMetric metric) { - KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); Meter currentMeter = meter; if (currentMeter == null) { - logger.log(Level.FINEST, "Metric changed but meter not set: {0}", kafkaMetricId); + logger.log(Level.FINEST, "Metric changed but meter not set: {0}", metric.metricName()); return; } RegisteredObservable registeredObservable = - KafkaMetricRegistry.getRegisteredObservable(currentMeter, kafkaMetricId, metric); + KafkaMetricRegistry.getRegisteredObservable(currentMeter, metric); if (registeredObservable == null) { - logger.log(Level.FINEST, "Metric changed but cannot map to instrument: {0}", kafkaMetricId); + logger.log( + Level.FINEST, "Metric changed but cannot map to instrument: {0}", metric.metricName()); return; } - InstrumentDescriptor instrumentDescriptor = registeredObservable.getInstrumentDescriptor(); - Attributes attributes = registeredObservable.getAttributes(); - Set> attributeKeys = attributes.asMap().keySet(); - - for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { - RegisteredObservable curRegisteredObservable = it.next(); - Set> curAttributeKeys = - curRegisteredObservable.getAttributes().asMap().keySet(); - if (curRegisteredObservable.equals(registeredObservable)) { - logger.log(Level.FINEST, "Replacing instrument: {0}", curRegisteredObservable); - closeInstrument(curRegisteredObservable.getObservable()); - it.remove(); - } else if (curRegisteredObservable.getInstrumentDescriptor().equals(instrumentDescriptor) - && attributeKeys.size() > curAttributeKeys.size() - && attributeKeys.containsAll(curAttributeKeys)) { - logger.log( - Level.FINEST, - "Replacing instrument with higher dimension version: {0}", - curRegisteredObservable); - closeInstrument(curRegisteredObservable.getObservable()); - it.remove(); + Set> attributeKeys = registeredObservable.getAttributes().asMap().keySet(); + synchronized (lock) { + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + RegisteredObservable curRegisteredObservable = it.next(); + Set> curAttributeKeys = + curRegisteredObservable.getAttributes().asMap().keySet(); + if (curRegisteredObservable.getKafkaMetricName().equals(metric.metricName())) { + logger.log(Level.FINEST, "Replacing instrument: {0}", curRegisteredObservable); + closeInstrument(curRegisteredObservable.getObservable()); + it.remove(); + } else if (curRegisteredObservable + .getInstrumentDescriptor() + .equals(registeredObservable.getInstrumentDescriptor()) + && attributeKeys.size() > curAttributeKeys.size() + && attributeKeys.containsAll(curAttributeKeys)) { + logger.log( + Level.FINEST, + "Replacing instrument with higher dimension version: {0}", + curRegisteredObservable); + closeInstrument(curRegisteredObservable.getObservable()); + it.remove(); + } } - } - registeredObservables.add(registeredObservable); + registeredObservables.add(registeredObservable); + } } @Override public void metricRemoval(KafkaMetric metric) { - KafkaMetricId kafkaMetricId = KafkaMetricId.create(metric); - logger.log(Level.FINEST, "Metric removed: {0}", kafkaMetricId); - for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { - RegisteredObservable current = it.next(); - if (current.matches(metric)) { - closeInstrument(current.getObservable()); - it.remove(); + logger.log(Level.FINEST, "Metric removed: {0}", metric.metricName()); + synchronized (lock) { + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + RegisteredObservable current = it.next(); + if (current.getKafkaMetricName().equals(metric.metricName())) { + closeInstrument(current.getObservable()); + it.remove(); + } } } } @@ -131,9 +138,11 @@ public void close() { } static void closeAllInstruments() { - for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { - closeInstrument(it.next().getObservable()); - it.remove(); + synchronized (lock) { + for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) { + closeInstrument(it.next().getObservable()); + it.remove(); + } } } diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java index 824d7c2cbc01..0ae8ded63f62 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java @@ -5,16 +5,14 @@ package io.opentelemetry.instrumentation.kafkaclients; -import static io.opentelemetry.instrumentation.kafkaclients.KafkaMetricRegistry.toAttributes; - import com.google.auto.value.AutoValue; import io.opentelemetry.api.common.Attributes; -import org.apache.kafka.common.metrics.KafkaMetric; +import org.apache.kafka.common.MetricName; @AutoValue abstract class RegisteredObservable { - abstract KafkaMetricId getKafkaMetricId(); + abstract MetricName getKafkaMetricName(); abstract InstrumentDescriptor getInstrumentDescriptor(); @@ -22,17 +20,12 @@ abstract class RegisteredObservable { abstract AutoCloseable getObservable(); - boolean matches(KafkaMetric kafkaMetric) { - return getKafkaMetricId().equals(KafkaMetricId.create(kafkaMetric)) - && getAttributes().equals(toAttributes(kafkaMetric)); - } - static RegisteredObservable create( - KafkaMetricId kafkaMetricId, + MetricName metricName, InstrumentDescriptor instrumentDescriptor, Attributes attributes, AutoCloseable observable) { return new AutoValue_RegisteredObservable( - kafkaMetricId, instrumentDescriptor, attributes, observable); + metricName, instrumentDescriptor, attributes, observable); } } From 071edb100ed1408006588cdb1168d89716fc3caa Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Thu, 23 Jun 2022 12:35:35 -0500 Subject: [PATCH 11/17] Remove synchornization from metricChange --- .../instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index a289071879d1..1467e14725b8 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -75,7 +75,7 @@ public void init(List metrics) { } @Override - public synchronized void metricChange(KafkaMetric metric) { + public void metricChange(KafkaMetric metric) { Meter currentMeter = meter; if (currentMeter == null) { logger.log(Level.FINEST, "Metric changed but meter not set: {0}", metric.metricName()); From 681534f5e98e5741d663751876691e667e2910a3 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Mon, 27 Jun 2022 12:26:42 -0500 Subject: [PATCH 12/17] remove kafka dependency --- .../kafka-clients/kafka-clients-0.11/testing/build.gradle.kts | 2 -- 1 file changed, 2 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts index 1bd73daac2aa..3058fd208b2f 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts @@ -14,6 +14,4 @@ dependencies { compileOnly("com.google.auto.value:auto-value-annotations") annotationProcessor("com.google.auto.value:auto-value") - - runtimeOnly("org.apache.kafka:kafka_2.13:2.8.1") } From a630bdbd8bad75e816d5a216304bfa32c39a9d00 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 28 Jun 2022 14:00:24 -0500 Subject: [PATCH 13/17] PR feedback --- .../OpenTelemetryKafkaMetricsTest.java | 11 +- .../kafka-clients-common/library/README.md | 412 +++++++++--------- .../OpenTelemetryKafkaMetrics.java | 26 +- 3 files changed, 216 insertions(+), 233 deletions(-) diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java index c5c640dcfc68..2d5676cadfef 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java @@ -77,8 +77,6 @@ abstract class OpenTelemetryKafkaMetricsTest { @BeforeEach void setup() { - OpenTelemetryKafkaMetrics.setOpenTelemetry(testing.getOpenTelemetry()); - String metricReporters = OpenTelemetryKafkaMetrics.class.getName() + "," + TestMetricsReporter.class.getName(); @@ -367,16 +365,15 @@ private static void printMappingTable() { StringBuilder sb = new StringBuilder(); // Append table headers sb.append( - "| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type |") + "| Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type |") .append(lineSeparator()) .append( - "|---|--------------|-------------|----------------|-----------------|------------------------|-----------------|") + "|--------------|-------------|----------------|-----------------|------------------------|-----------------|") .append(lineSeparator()); Map> kafkaMetricsByGroup = TestMetricsReporter.seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup)); List registeredObservables = OpenTelemetryKafkaMetrics.getRegisteredObservables(); - int count = 1; // Iterate through groups in alpha order for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) { List kafkaMetricIds = @@ -399,8 +396,7 @@ private static void printMappingTable() { // Append table row sb.append( String.format( - "| %s | %s | %s | %s | %s | %s | %s |%n", - count, + "| %s | %s | %s | %s | %s | %s |%n", "`" + group + "`", "`" + kafkaMetricId.getName() + "`", kafkaMetricId.getAttributeKeys().stream() @@ -409,7 +405,6 @@ private static void printMappingTable() { descriptor.map(i -> "`" + i.getName() + "`").orElse(""), descriptor.map(InstrumentDescriptor::getDescription).orElse(""), descriptor.map(i -> "`" + i.getInstrumentType() + "`").orElse(""))); - count++; } } logger.info("Mapping table" + System.lineSeparator() + sb); diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md index eb8502d00777..ed2610e58a1e 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md @@ -3,8 +3,8 @@ The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface. OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry. -To use, configure `OpenTelemetryKafkaMetrics` with an OpenTelemetry instance -via `OpenTelemetryKafkaMetrics#setOpenTelemetry(OpenTelemetry)`, and include a reference to this +To use, configure `GlobalOpenTelemetry` with an OpenTelemetry instance +via `GlobalOpenTelemetry#set(OpenTelemetry)`, and include a reference to this class in kafka producer or consumer configuration, i.e.: ```java @@ -27,207 +27,207 @@ OpenTelemetry metric each maps to (if available). Empty values in the Instrument Description, etc column indicates there is no registered mapping for the metric and data is NOT collected. -| # | Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | -|---|--------------|-------------|----------------|-----------------|------------------------|-----------------| -| 1 | `app-info` | `commit-id` | `client-id` | | | | -| 2 | `app-info` | `start-time-ms` | `client-id` | | | | -| 3 | `app-info` | `version` | `client-id` | | | | -| 4 | `consumer-coordinator-metrics` | `assigned-partitions` | `client-id` | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | -| 5 | `consumer-coordinator-metrics` | `commit-latency-avg` | `client-id` | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | -| 6 | `consumer-coordinator-metrics` | `commit-latency-max` | `client-id` | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | -| 7 | `consumer-coordinator-metrics` | `commit-rate` | `client-id` | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 8 | `consumer-coordinator-metrics` | `commit-total` | `client-id` | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | -| 9 | `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | `client-id` | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_GAUGE` | -| 10 | `consumer-coordinator-metrics` | `failed-rebalance-total` | `client-id` | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | -| 11 | `consumer-coordinator-metrics` | `heartbeat-rate` | `client-id` | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 12 | `consumer-coordinator-metrics` | `heartbeat-response-time-max` | `client-id` | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | -| 13 | `consumer-coordinator-metrics` | `heartbeat-total` | `client-id` | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | -| 14 | `consumer-coordinator-metrics` | `join-rate` | `client-id` | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 15 | `consumer-coordinator-metrics` | `join-time-avg` | `client-id` | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | -| 16 | `consumer-coordinator-metrics` | `join-time-max` | `client-id` | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | -| 17 | `consumer-coordinator-metrics` | `join-total` | `client-id` | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | -| 18 | `consumer-coordinator-metrics` | `last-heartbeat-seconds-ago` | `client-id` | `kafka.consumer.last_heartbeat_seconds_ago` | The number of seconds since the last coordinator heartbeat was sent | `DOUBLE_OBSERVABLE_GAUGE` | -| 19 | `consumer-coordinator-metrics` | `last-rebalance-seconds-ago` | `client-id` | `kafka.consumer.last_rebalance_seconds_ago` | The number of seconds since the last successful rebalance event | `DOUBLE_OBSERVABLE_GAUGE` | -| 20 | `consumer-coordinator-metrics` | `partition-assigned-latency-avg` | `client-id` | `kafka.consumer.partition_assigned_latency_avg` | The average time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 21 | `consumer-coordinator-metrics` | `partition-assigned-latency-max` | `client-id` | `kafka.consumer.partition_assigned_latency_max` | The max time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 22 | `consumer-coordinator-metrics` | `partition-lost-latency-avg` | `client-id` | `kafka.consumer.partition_lost_latency_avg` | The average time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 23 | `consumer-coordinator-metrics` | `partition-lost-latency-max` | `client-id` | `kafka.consumer.partition_lost_latency_max` | The max time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 24 | `consumer-coordinator-metrics` | `partition-revoked-latency-avg` | `client-id` | `kafka.consumer.partition_revoked_latency_avg` | The average time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 25 | `consumer-coordinator-metrics` | `partition-revoked-latency-max` | `client-id` | `kafka.consumer.partition_revoked_latency_max` | The max time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | -| 26 | `consumer-coordinator-metrics` | `rebalance-latency-avg` | `client-id` | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 27 | `consumer-coordinator-metrics` | `rebalance-latency-max` | `client-id` | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 28 | `consumer-coordinator-metrics` | `rebalance-latency-total` | `client-id` | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | -| 29 | `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | `client-id` | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | -| 30 | `consumer-coordinator-metrics` | `rebalance-total` | `client-id` | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | -| 31 | `consumer-coordinator-metrics` | `sync-rate` | `client-id` | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 32 | `consumer-coordinator-metrics` | `sync-time-avg` | `client-id` | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | -| 33 | `consumer-coordinator-metrics` | `sync-time-max` | `client-id` | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | -| 34 | `consumer-coordinator-metrics` | `sync-total` | `client-id` | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | -| 35 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id` | | | | -| 36 | `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 37 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id` | | | | -| 38 | `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 39 | `consumer-fetch-manager-metrics` | `fetch-latency-avg` | `client-id` | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 40 | `consumer-fetch-manager-metrics` | `fetch-latency-max` | `client-id` | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 41 | `consumer-fetch-manager-metrics` | `fetch-rate` | `client-id` | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 42 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id` | | | | -| 43 | `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id`,`topic` | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 44 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id` | | | | -| 45 | `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id`,`topic` | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | -| 46 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | `client-id` | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 47 | `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | `client-id` | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 48 | `consumer-fetch-manager-metrics` | `fetch-total` | `client-id` | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | -| 49 | `consumer-fetch-manager-metrics` | `preferred-read-replica` | `client-id`,`topic`,`partition` | | | | -| 50 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id` | | | | -| 51 | `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id`,`topic` | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 52 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id` | | | | -| 53 | `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id`,`topic` | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | -| 54 | `consumer-fetch-manager-metrics` | `records-lag` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 55 | `consumer-fetch-manager-metrics` | `records-lag-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 56 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id` | | | | -| 57 | `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 58 | `consumer-fetch-manager-metrics` | `records-lead` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 59 | `consumer-fetch-manager-metrics` | `records-lead-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | -| 60 | `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id` | | | | -| 61 | `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | -| 62 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id` | | | | -| 63 | `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id`,`topic` | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | -| 64 | `consumer-metrics` | `connection-close-rate` | `client-id` | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 65 | `consumer-metrics` | `connection-close-total` | `client-id` | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | -| 66 | `consumer-metrics` | `connection-count` | `client-id` | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 67 | `consumer-metrics` | `connection-creation-rate` | `client-id` | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 68 | `consumer-metrics` | `connection-creation-total` | `client-id` | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 69 | `consumer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 70 | `consumer-metrics` | `failed-authentication-total` | `client-id` | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 71 | `consumer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 72 | `consumer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 73 | `consumer-metrics` | `incoming-byte-rate` | `client-id` | | | | -| 74 | `consumer-metrics` | `incoming-byte-total` | `client-id` | | | | -| 75 | `consumer-metrics` | `io-ratio` | `client-id` | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | -| 76 | `consumer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 77 | `consumer-metrics` | `io-wait-ratio` | `client-id` | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | -| 78 | `consumer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 79 | `consumer-metrics` | `io-waittime-total` | `client-id` | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | -| 80 | `consumer-metrics` | `iotime-total` | `client-id` | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | -| 81 | `consumer-metrics` | `last-poll-seconds-ago` | `client-id` | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | -| 82 | `consumer-metrics` | `network-io-rate` | `client-id` | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 83 | `consumer-metrics` | `network-io-total` | `client-id` | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 84 | `consumer-metrics` | `outgoing-byte-rate` | `client-id` | | | | -| 85 | `consumer-metrics` | `outgoing-byte-total` | `client-id` | | | | -| 86 | `consumer-metrics` | `poll-idle-ratio-avg` | `client-id` | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | -| 87 | `consumer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 88 | `consumer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 89 | `consumer-metrics` | `request-rate` | `client-id` | | | | -| 90 | `consumer-metrics` | `request-size-avg` | `client-id` | | | | -| 91 | `consumer-metrics` | `request-size-max` | `client-id` | | | | -| 92 | `consumer-metrics` | `request-total` | `client-id` | | | | -| 93 | `consumer-metrics` | `response-rate` | `client-id` | | | | -| 94 | `consumer-metrics` | `response-total` | `client-id` | | | | -| 95 | `consumer-metrics` | `select-rate` | `client-id` | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 96 | `consumer-metrics` | `select-total` | `client-id` | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | -| 97 | `consumer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 98 | `consumer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 99 | `consumer-metrics` | `successful-authentication-total` | `client-id` | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 100 | `consumer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 101 | `consumer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 102 | `consumer-metrics` | `time-between-poll-avg` | `client-id` | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | -| 103 | `consumer-metrics` | `time-between-poll-max` | `client-id` | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | -| 104 | `consumer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 105 | `consumer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 106 | `consumer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 107 | `consumer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 108 | `consumer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | -| 109 | `consumer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | -| 110 | `consumer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 111 | `consumer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 112 | `consumer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 113 | `consumer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 114 | `consumer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 115 | `consumer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 116 | `kafka-metrics-count` | `count` | `client-id` | | | | -| 117 | `producer-metrics` | `batch-size-avg` | `client-id` | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 118 | `producer-metrics` | `batch-size-max` | `client-id` | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 119 | `producer-metrics` | `batch-split-rate` | `client-id` | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 120 | `producer-metrics` | `batch-split-total` | `client-id` | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | -| 121 | `producer-metrics` | `buffer-available-bytes` | `client-id` | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | -| 122 | `producer-metrics` | `buffer-exhausted-rate` | `client-id` | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_GAUGE` | -| 123 | `producer-metrics` | `buffer-exhausted-total` | `client-id` | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | -| 124 | `producer-metrics` | `buffer-total-bytes` | `client-id` | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | -| 125 | `producer-metrics` | `bufferpool-wait-ratio` | `client-id` | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_GAUGE` | -| 126 | `producer-metrics` | `bufferpool-wait-time-total` | `client-id` | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | -| 127 | `producer-metrics` | `compression-rate-avg` | `client-id` | `kafka.producer.compression_rate_avg` | The average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 128 | `producer-metrics` | `connection-close-rate` | `client-id` | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 129 | `producer-metrics` | `connection-close-total` | `client-id` | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | -| 130 | `producer-metrics` | `connection-count` | `client-id` | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | -| 131 | `producer-metrics` | `connection-creation-rate` | `client-id` | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 132 | `producer-metrics` | `connection-creation-total` | `client-id` | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | -| 133 | `producer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 134 | `producer-metrics` | `failed-authentication-total` | `client-id` | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 135 | `producer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 136 | `producer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 137 | `producer-metrics` | `incoming-byte-rate` | `client-id` | | | | -| 138 | `producer-metrics` | `incoming-byte-total` | `client-id` | | | | -| 139 | `producer-metrics` | `io-ratio` | `client-id` | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | -| 140 | `producer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 141 | `producer-metrics` | `io-wait-ratio` | `client-id` | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | -| 142 | `producer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | -| 143 | `producer-metrics` | `io-waittime-total` | `client-id` | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | -| 144 | `producer-metrics` | `iotime-total` | `client-id` | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | -| 145 | `producer-metrics` | `metadata-age` | `client-id` | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | -| 146 | `producer-metrics` | `network-io-rate` | `client-id` | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 147 | `producer-metrics` | `network-io-total` | `client-id` | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 148 | `producer-metrics` | `outgoing-byte-rate` | `client-id` | | | | -| 149 | `producer-metrics` | `outgoing-byte-total` | `client-id` | | | | -| 150 | `producer-metrics` | `produce-throttle-time-avg` | `client-id` | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | -| 151 | `producer-metrics` | `produce-throttle-time-max` | `client-id` | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | -| 152 | `producer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 153 | `producer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | -| 154 | `producer-metrics` | `record-error-rate` | `client-id` | | | | -| 155 | `producer-metrics` | `record-error-total` | `client-id` | | | | -| 156 | `producer-metrics` | `record-queue-time-avg` | `client-id` | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | -| 157 | `producer-metrics` | `record-queue-time-max` | `client-id` | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | -| 158 | `producer-metrics` | `record-retry-rate` | `client-id` | | | | -| 159 | `producer-metrics` | `record-retry-total` | `client-id` | | | | -| 160 | `producer-metrics` | `record-send-rate` | `client-id` | | | | -| 161 | `producer-metrics` | `record-send-total` | `client-id` | | | | -| 162 | `producer-metrics` | `record-size-avg` | `client-id` | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | -| 163 | `producer-metrics` | `record-size-max` | `client-id` | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | -| 164 | `producer-metrics` | `records-per-request-avg` | `client-id` | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | -| 165 | `producer-metrics` | `request-latency-avg` | `client-id` | | | | -| 166 | `producer-metrics` | `request-latency-max` | `client-id` | | | | -| 167 | `producer-metrics` | `request-rate` | `client-id` | | | | -| 168 | `producer-metrics` | `request-size-avg` | `client-id` | | | | -| 169 | `producer-metrics` | `request-size-max` | `client-id` | | | | -| 170 | `producer-metrics` | `request-total` | `client-id` | | | | -| 171 | `producer-metrics` | `requests-in-flight` | `client-id` | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | -| 172 | `producer-metrics` | `response-rate` | `client-id` | | | | -| 173 | `producer-metrics` | `response-total` | `client-id` | | | | -| 174 | `producer-metrics` | `select-rate` | `client-id` | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 175 | `producer-metrics` | `select-total` | `client-id` | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | -| 176 | `producer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 177 | `producer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 178 | `producer-metrics` | `successful-authentication-total` | `client-id` | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | -| 179 | `producer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 180 | `producer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | -| 181 | `producer-metrics` | `waiting-threads` | `client-id` | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | -| 182 | `producer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 183 | `producer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | -| 184 | `producer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 185 | `producer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | -| 186 | `producer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 187 | `producer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | -| 188 | `producer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 189 | `producer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 190 | `producer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | -| 191 | `producer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | -| 192 | `producer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | -| 193 | `producer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | -| 194 | `producer-topic-metrics` | `byte-rate` | `client-id`,`topic` | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | -| 195 | `producer-topic-metrics` | `byte-total` | `client-id`,`topic` | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | -| 196 | `producer-topic-metrics` | `compression-rate` | `client-id`,`topic` | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size. | `DOUBLE_OBSERVABLE_GAUGE` | -| 197 | `producer-topic-metrics` | `record-error-rate` | `client-id`,`topic` | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | -| 198 | `producer-topic-metrics` | `record-error-total` | `client-id`,`topic` | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | -| 199 | `producer-topic-metrics` | `record-retry-rate` | `client-id`,`topic` | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | -| 200 | `producer-topic-metrics` | `record-retry-total` | `client-id`,`topic` | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | -| 201 | `producer-topic-metrics` | `record-send-rate` | `client-id`,`topic` | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | -| 202 | `producer-topic-metrics` | `record-send-total` | `client-id`,`topic` | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | +| Metric Group | Metric Name | Attribute Keys | Instrument Name | Instrument Description | Instrument Type | +|--------------|-------------|----------------|-----------------|------------------------|-----------------| +| `app-info` | `commit-id` | `client-id` | | | | +| `app-info` | `start-time-ms` | `client-id` | | | | +| `app-info` | `version` | `client-id` | | | | +| `consumer-coordinator-metrics` | `assigned-partitions` | `client-id` | `kafka.consumer.assigned_partitions` | The number of partitions currently assigned to this consumer | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `commit-latency-avg` | `client-id` | `kafka.consumer.commit_latency_avg` | The average time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `commit-latency-max` | `client-id` | `kafka.consumer.commit_latency_max` | The max time taken for a commit request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `commit-rate` | `client-id` | `kafka.consumer.commit_rate` | The number of commit calls per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `commit-total` | `client-id` | `kafka.consumer.commit_total` | The total number of commit calls | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `failed-rebalance-rate-per-hour` | `client-id` | `kafka.consumer.failed_rebalance_rate_per_hour` | The number of failed rebalance events per hour | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `failed-rebalance-total` | `client-id` | `kafka.consumer.failed_rebalance_total` | The total number of failed rebalance events | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `heartbeat-rate` | `client-id` | `kafka.consumer.heartbeat_rate` | The number of heartbeats per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `heartbeat-response-time-max` | `client-id` | `kafka.consumer.heartbeat_response_time_max` | The max time taken to receive a response to a heartbeat request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `heartbeat-total` | `client-id` | `kafka.consumer.heartbeat_total` | The total number of heartbeats | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `join-rate` | `client-id` | `kafka.consumer.join_rate` | The number of group joins per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `join-time-avg` | `client-id` | `kafka.consumer.join_time_avg` | The average time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `join-time-max` | `client-id` | `kafka.consumer.join_time_max` | The max time taken for a group rejoin | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `join-total` | `client-id` | `kafka.consumer.join_total` | The total number of group joins | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `last-heartbeat-seconds-ago` | `client-id` | `kafka.consumer.last_heartbeat_seconds_ago` | The number of seconds since the last coordinator heartbeat was sent | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `last-rebalance-seconds-ago` | `client-id` | `kafka.consumer.last_rebalance_seconds_ago` | The number of seconds since the last successful rebalance event | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-assigned-latency-avg` | `client-id` | `kafka.consumer.partition_assigned_latency_avg` | The average time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-assigned-latency-max` | `client-id` | `kafka.consumer.partition_assigned_latency_max` | The max time taken for a partition-assigned rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-lost-latency-avg` | `client-id` | `kafka.consumer.partition_lost_latency_avg` | The average time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-lost-latency-max` | `client-id` | `kafka.consumer.partition_lost_latency_max` | The max time taken for a partition-lost rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-revoked-latency-avg` | `client-id` | `kafka.consumer.partition_revoked_latency_avg` | The average time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `partition-revoked-latency-max` | `client-id` | `kafka.consumer.partition_revoked_latency_max` | The max time taken for a partition-revoked rebalance listener callback | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `rebalance-latency-avg` | `client-id` | `kafka.consumer.rebalance_latency_avg` | The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `rebalance-latency-max` | `client-id` | `kafka.consumer.rebalance_latency_max` | The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `rebalance-latency-total` | `client-id` | `kafka.consumer.rebalance_latency_total` | The total number of milliseconds this consumer has spent in successful rebalances since creation | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `rebalance-rate-per-hour` | `client-id` | `kafka.consumer.rebalance_rate_per_hour` | The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `rebalance-total` | `client-id` | `kafka.consumer.rebalance_total` | The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-coordinator-metrics` | `sync-rate` | `client-id` | `kafka.consumer.sync_rate` | The number of group syncs per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `sync-time-avg` | `client-id` | `kafka.consumer.sync_time_avg` | The average time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `sync-time-max` | `client-id` | `kafka.consumer.sync_time_max` | The max time taken for a group sync | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-coordinator-metrics` | `sync-total` | `client-id` | `kafka.consumer.sync_total` | The total number of group syncs | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `bytes-consumed-rate` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_rate` | The average number of bytes consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `bytes-consumed-total` | `client-id`,`topic` | `kafka.consumer.bytes_consumed_total` | The total number of bytes consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-fetch-manager-metrics` | `fetch-latency-avg` | `client-id` | `kafka.consumer.fetch_latency_avg` | The average time taken for a fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-latency-max` | `client-id` | `kafka.consumer.fetch_latency_max` | The max time taken for any fetch request. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-rate` | `client-id` | `kafka.consumer.fetch_rate` | The number of fetch requests per second. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `fetch-size-avg` | `client-id`,`topic` | `kafka.consumer.fetch_size_avg` | The average number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `fetch-size-max` | `client-id`,`topic` | `kafka.consumer.fetch_size_max` | The maximum number of bytes fetched per request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-throttle-time-avg` | `client-id` | `kafka.consumer.fetch_throttle_time_avg` | The average throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-throttle-time-max` | `client-id` | `kafka.consumer.fetch_throttle_time_max` | The maximum throttle time in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `fetch-total` | `client-id` | `kafka.consumer.fetch_total` | The total number of fetch requests. | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-fetch-manager-metrics` | `preferred-read-replica` | `client-id`,`topic`,`partition` | | | | +| `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `records-consumed-rate` | `client-id`,`topic` | `kafka.consumer.records_consumed_rate` | The average number of records consumed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `records-consumed-total` | `client-id`,`topic` | `kafka.consumer.records_consumed_total` | The total number of records consumed | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-fetch-manager-metrics` | `records-lag` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag` | The latest lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-lag-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_avg` | The average lag of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `records-lag-max` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lag_max` | The maximum lag in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-lead` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead` | The latest lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-lead-avg` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_avg` | The average lead of the partition | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `records-lead-min` | `client-id`,`topic`,`partition` | `kafka.consumer.records_lead_min` | The minimum lead in terms of number of records for any partition in this window | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id` | | | | +| `consumer-fetch-manager-metrics` | `records-per-request-avg` | `client-id`,`topic` | `kafka.consumer.records_per_request_avg` | The average number of records in each request | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `connection-close-rate` | `client-id` | `kafka.consumer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `connection-close-total` | `client-id` | `kafka.consumer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `connection-count` | `client-id` | `kafka.consumer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `connection-creation-rate` | `client-id` | `kafka.consumer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `connection-creation-total` | `client-id` | `kafka.consumer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.consumer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `failed-authentication-total` | `client-id` | `kafka.consumer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.consumer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.consumer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `incoming-byte-rate` | `client-id` | | | | +| `consumer-metrics` | `incoming-byte-total` | `client-id` | | | | +| `consumer-metrics` | `io-ratio` | `client-id` | `kafka.consumer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.consumer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `io-wait-ratio` | `client-id` | `kafka.consumer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.consumer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `io-waittime-total` | `client-id` | `kafka.consumer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `iotime-total` | `client-id` | `kafka.consumer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `last-poll-seconds-ago` | `client-id` | `kafka.consumer.last_poll_seconds_ago` | The number of seconds since the last poll() invocation. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `network-io-rate` | `client-id` | `kafka.consumer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `network-io-total` | `client-id` | `kafka.consumer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `outgoing-byte-rate` | `client-id` | | | | +| `consumer-metrics` | `outgoing-byte-total` | `client-id` | | | | +| `consumer-metrics` | `poll-idle-ratio-avg` | `client-id` | `kafka.consumer.poll_idle_ratio_avg` | The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.consumer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.consumer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `request-rate` | `client-id` | | | | +| `consumer-metrics` | `request-size-avg` | `client-id` | | | | +| `consumer-metrics` | `request-size-max` | `client-id` | | | | +| `consumer-metrics` | `request-total` | `client-id` | | | | +| `consumer-metrics` | `response-rate` | `client-id` | | | | +| `consumer-metrics` | `response-total` | `client-id` | | | | +| `consumer-metrics` | `select-rate` | `client-id` | `kafka.consumer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `select-total` | `client-id` | `kafka.consumer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.consumer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.consumer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `successful-authentication-total` | `client-id` | `kafka.consumer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.consumer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.consumer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-metrics` | `time-between-poll-avg` | `client-id` | `kafka.consumer.time_between_poll_avg` | The average delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-metrics` | `time-between-poll-max` | `client-id` | `kafka.consumer.time_between_poll_max` | The max delay between invocations of poll(). | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.consumer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.consumer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.consumer.request_latency_avg` | | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.consumer.request_latency_max` | | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.consumer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.consumer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.consumer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.consumer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| `consumer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.consumer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `consumer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.consumer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| `kafka-metrics-count` | `count` | `client-id` | | | | +| `producer-metrics` | `batch-size-avg` | `client-id` | `kafka.producer.batch_size_avg` | The average number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `batch-size-max` | `client-id` | `kafka.producer.batch_size_max` | The max number of bytes sent per partition per-request. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `batch-split-rate` | `client-id` | `kafka.producer.batch_split_rate` | The average number of batch splits per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `batch-split-total` | `client-id` | `kafka.producer.batch_split_total` | The total number of batch splits | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `buffer-available-bytes` | `client-id` | `kafka.producer.buffer_available_bytes` | The total amount of buffer memory that is not being used (either unallocated or in the free list). | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `buffer-exhausted-rate` | `client-id` | `kafka.producer.buffer_exhausted_rate` | The average per-second number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `buffer-exhausted-total` | `client-id` | `kafka.producer.buffer_exhausted_total` | The total number of record sends that are dropped due to buffer exhaustion | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `buffer-total-bytes` | `client-id` | `kafka.producer.buffer_total_bytes` | The maximum amount of buffer memory the client can use (whether or not it is currently used). | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `bufferpool-wait-ratio` | `client-id` | `kafka.producer.bufferpool_wait_ratio` | The fraction of time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `bufferpool-wait-time-total` | `client-id` | `kafka.producer.bufferpool_wait_time_total` | The total time an appender waits for space allocation. | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `compression-rate-avg` | `client-id` | `kafka.producer.compression_rate_avg` | The average compression rate of record batches. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `connection-close-rate` | `client-id` | `kafka.producer.connection_close_rate` | The number of connections closed per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `connection-close-total` | `client-id` | `kafka.producer.connection_close_total` | The total number of connections closed | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `connection-count` | `client-id` | `kafka.producer.connection_count` | The current number of active connections. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `connection-creation-rate` | `client-id` | `kafka.producer.connection_creation_rate` | The number of new connections established per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `connection-creation-total` | `client-id` | `kafka.producer.connection_creation_total` | The total number of new connections established | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `failed-authentication-rate` | `client-id` | `kafka.producer.failed_authentication_rate` | The number of connections with failed authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `failed-authentication-total` | `client-id` | `kafka.producer.failed_authentication_total` | The total number of connections with failed authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `failed-reauthentication-rate` | `client-id` | `kafka.producer.failed_reauthentication_rate` | The number of failed re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `failed-reauthentication-total` | `client-id` | `kafka.producer.failed_reauthentication_total` | The total number of failed re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `incoming-byte-rate` | `client-id` | | | | +| `producer-metrics` | `incoming-byte-total` | `client-id` | | | | +| `producer-metrics` | `io-ratio` | `client-id` | `kafka.producer.io_ratio` | The fraction of time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `io-time-ns-avg` | `client-id` | `kafka.producer.io_time_ns_avg` | The average length of time for I/O per select call in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `io-wait-ratio` | `client-id` | `kafka.producer.io_wait_ratio` | The fraction of time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `io-wait-time-ns-avg` | `client-id` | `kafka.producer.io_wait_time_ns_avg` | The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `io-waittime-total` | `client-id` | `kafka.producer.io_waittime_total` | The total time the I/O thread spent waiting | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `iotime-total` | `client-id` | `kafka.producer.iotime_total` | The total time the I/O thread spent doing I/O | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `metadata-age` | `client-id` | `kafka.producer.metadata_age` | The age in seconds of the current producer metadata being used. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `network-io-rate` | `client-id` | `kafka.producer.network_io_rate` | The number of network operations (reads or writes) on all connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `network-io-total` | `client-id` | `kafka.producer.network_io_total` | The total number of network operations (reads or writes) on all connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `outgoing-byte-rate` | `client-id` | | | | +| `producer-metrics` | `outgoing-byte-total` | `client-id` | | | | +| `producer-metrics` | `produce-throttle-time-avg` | `client-id` | `kafka.producer.produce_throttle_time_avg` | The average time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `produce-throttle-time-max` | `client-id` | `kafka.producer.produce_throttle_time_max` | The maximum time in ms a request was throttled by a broker | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `reauthentication-latency-avg` | `client-id` | `kafka.producer.reauthentication_latency_avg` | The average latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `reauthentication-latency-max` | `client-id` | `kafka.producer.reauthentication_latency_max` | The max latency observed due to re-authentication | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `record-error-rate` | `client-id` | | | | +| `producer-metrics` | `record-error-total` | `client-id` | | | | +| `producer-metrics` | `record-queue-time-avg` | `client-id` | `kafka.producer.record_queue_time_avg` | The average time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `record-queue-time-max` | `client-id` | `kafka.producer.record_queue_time_max` | The maximum time in ms record batches spent in the send buffer. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `record-retry-rate` | `client-id` | | | | +| `producer-metrics` | `record-retry-total` | `client-id` | | | | +| `producer-metrics` | `record-send-rate` | `client-id` | | | | +| `producer-metrics` | `record-send-total` | `client-id` | | | | +| `producer-metrics` | `record-size-avg` | `client-id` | `kafka.producer.record_size_avg` | The average record size | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `record-size-max` | `client-id` | `kafka.producer.record_size_max` | The maximum record size | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `records-per-request-avg` | `client-id` | `kafka.producer.records_per_request_avg` | The average number of records per request. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `request-latency-avg` | `client-id` | | | | +| `producer-metrics` | `request-latency-max` | `client-id` | | | | +| `producer-metrics` | `request-rate` | `client-id` | | | | +| `producer-metrics` | `request-size-avg` | `client-id` | | | | +| `producer-metrics` | `request-size-max` | `client-id` | | | | +| `producer-metrics` | `request-total` | `client-id` | | | | +| `producer-metrics` | `requests-in-flight` | `client-id` | `kafka.producer.requests_in_flight` | The current number of in-flight requests awaiting a response. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `response-rate` | `client-id` | | | | +| `producer-metrics` | `response-total` | `client-id` | | | | +| `producer-metrics` | `select-rate` | `client-id` | `kafka.producer.select_rate` | The number of times the I/O layer checked for new I/O to perform per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `select-total` | `client-id` | `kafka.producer.select_total` | The total number of times the I/O layer checked for new I/O to perform | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `successful-authentication-no-reauth-total` | `client-id` | `kafka.producer.successful_authentication_no_reauth_total` | The total number of connections with successful authentication where the client does not support re-authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `successful-authentication-rate` | `client-id` | `kafka.producer.successful_authentication_rate` | The number of connections with successful authentication per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `successful-authentication-total` | `client-id` | `kafka.producer.successful_authentication_total` | The total number of connections with successful authentication | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `successful-reauthentication-rate` | `client-id` | `kafka.producer.successful_reauthentication_rate` | The number of successful re-authentication of connections per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-metrics` | `successful-reauthentication-total` | `client-id` | `kafka.producer.successful_reauthentication_total` | The total number of successful re-authentication of connections | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-metrics` | `waiting-threads` | `client-id` | `kafka.producer.waiting_threads` | The number of user threads blocked waiting for buffer memory to enqueue their records | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `incoming-byte-rate` | `client-id`,`node-id` | `kafka.producer.incoming_byte_rate` | The number of bytes read off all sockets per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `incoming-byte-total` | `client-id`,`node-id` | `kafka.producer.incoming_byte_total` | The total number of bytes read off all sockets | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-node-metrics` | `outgoing-byte-rate` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_rate` | The number of outgoing bytes sent to all servers per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `outgoing-byte-total` | `client-id`,`node-id` | `kafka.producer.outgoing_byte_total` | The total number of outgoing bytes sent to all servers | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-node-metrics` | `request-latency-avg` | `client-id`,`node-id` | `kafka.producer.request_latency_avg` | The average request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `request-latency-max` | `client-id`,`node-id` | `kafka.producer.request_latency_max` | The maximum request latency in ms | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `request-rate` | `client-id`,`node-id` | `kafka.producer.request_rate` | The number of requests sent per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `request-size-avg` | `client-id`,`node-id` | `kafka.producer.request_size_avg` | The average size of requests sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `request-size-max` | `client-id`,`node-id` | `kafka.producer.request_size_max` | The maximum size of any request sent. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `request-total` | `client-id`,`node-id` | `kafka.producer.request_total` | The total number of requests sent | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-node-metrics` | `response-rate` | `client-id`,`node-id` | `kafka.producer.response_rate` | The number of responses received per second | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-node-metrics` | `response-total` | `client-id`,`node-id` | `kafka.producer.response_total` | The total number of responses received | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-topic-metrics` | `byte-rate` | `client-id`,`topic` | `kafka.producer.byte_rate` | The average number of bytes sent per second for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-topic-metrics` | `byte-total` | `client-id`,`topic` | `kafka.producer.byte_total` | The total number of bytes sent for a topic. | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-topic-metrics` | `compression-rate` | `client-id`,`topic` | `kafka.producer.compression_rate` | The average compression rate of record batches for a topic. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-topic-metrics` | `record-error-rate` | `client-id`,`topic` | `kafka.producer.record_error_rate` | The average per-second number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-topic-metrics` | `record-error-total` | `client-id`,`topic` | `kafka.producer.record_error_total` | The total number of record sends that resulted in errors | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-topic-metrics` | `record-retry-rate` | `client-id`,`topic` | `kafka.producer.record_retry_rate` | The average per-second number of retried record sends | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-topic-metrics` | `record-retry-total` | `client-id`,`topic` | `kafka.producer.record_retry_total` | The total number of retried record sends | `DOUBLE_OBSERVABLE_COUNTER` | +| `producer-topic-metrics` | `record-send-rate` | `client-id`,`topic` | `kafka.producer.record_send_rate` | The average number of records sent per second. | `DOUBLE_OBSERVABLE_GAUGE` | +| `producer-topic-metrics` | `record-send-total` | `client-id`,`topic` | `kafka.producer.record_send_total` | The total number of records sent. | `DOUBLE_OBSERVABLE_COUNTER` | diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java index 1467e14725b8..1c9d98d806ae 100644 --- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java +++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java @@ -5,6 +5,7 @@ package io.opentelemetry.instrumentation.kafkaclients; +import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.OpenTelemetry; import io.opentelemetry.api.common.AttributeKey; import io.opentelemetry.api.metrics.Meter; @@ -16,15 +17,15 @@ import java.util.Set; import java.util.logging.Level; import java.util.logging.Logger; -import javax.annotation.Nullable; import org.apache.kafka.common.metrics.KafkaMetric; import org.apache.kafka.common.metrics.MetricsReporter; /** * A {@link MetricsReporter} which bridges Kafka metrics to OpenTelemetry metrics. * - *

To use, configure OpenTelemetry instance via {@link #setOpenTelemetry(OpenTelemetry)}, and - * include a reference to this class in kafka producer or consumer configuration, i.e.: + *

To use, configure {@link GlobalOpenTelemetry} instance via {@link + * GlobalOpenTelemetry#set(OpenTelemetry)}, and include a reference to this class in kafka producer + * or consumer configuration, i.e.: * * 

{@code
  * //    Map config = new HashMap<>();
@@ -38,21 +39,14 @@
 public class OpenTelemetryKafkaMetrics implements MetricsReporter {
 
   private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName());
-  @Nullable private static volatile Meter meter;
+  private static volatile Meter meter =
+      GlobalOpenTelemetry.getMeter("io.opentelemetry.kafka-clients");
 
   private static final Object lock = new Object();
 
   @GuardedBy("lock")
   private static final List registeredObservables = new ArrayList<>();
 
-  /**
-   * Setup OpenTelemetry. This should be called as early in the application lifecycle as possible.
-   * Kafka metrics that are observed before this is called may not be bridged.
-   */
-  public static void setOpenTelemetry(OpenTelemetry openTelemetry) {
-    meter = openTelemetry.getMeter("io.opentelemetry.kafka-clients");
-  }
-
   /**
    * Reset for test by reseting the {@link #meter} to {@code null} and closing all registered
    * instruments.
@@ -76,14 +70,8 @@ public void init(List metrics) {
 
   @Override
   public void metricChange(KafkaMetric metric) {
-    Meter currentMeter = meter;
-    if (currentMeter == null) {
-      logger.log(Level.FINEST, "Metric changed but meter not set: {0}", metric.metricName());
-      return;
-    }
-
     RegisteredObservable registeredObservable =
-        KafkaMetricRegistry.getRegisteredObservable(currentMeter, metric);
+        KafkaMetricRegistry.getRegisteredObservable(meter, metric);
     if (registeredObservable == null) {
       logger.log(
           Level.FINEST, "Metric changed but cannot map to instrument: {0}", metric.metricName());

From 3c6b04d8e4df7a48a8ceaa3409ed298140ded6f2 Mon Sep 17 00:00:00 2001
From: Jack Berg 
Date: Thu, 30 Jun 2022 10:08:51 -0500
Subject: [PATCH 14/17] Fix reset

---
 .../kafkaclients/OpenTelemetryKafkaMetrics.java       | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
index 1c9d98d806ae..857ab52d681e 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
@@ -39,8 +39,7 @@
 public class OpenTelemetryKafkaMetrics implements MetricsReporter {
 
   private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName());
-  private static volatile Meter meter =
-      GlobalOpenTelemetry.getMeter("io.opentelemetry.kafka-clients");
+  private static volatile Meter meter;
 
   private static final Object lock = new Object();
 
@@ -70,8 +69,14 @@ public void init(List metrics) {
 
   @Override
   public void metricChange(KafkaMetric metric) {
+    // Maybe initialize meter
+    Meter currentMeter = meter;
+    if (currentMeter == null) {
+      currentMeter = meter = GlobalOpenTelemetry.getMeter("io.opentelemetry.kafka-clients");
+    }
+
     RegisteredObservable registeredObservable =
-        KafkaMetricRegistry.getRegisteredObservable(meter, metric);
+        KafkaMetricRegistry.getRegisteredObservable(currentMeter, metric);
     if (registeredObservable == null) {
       logger.log(
           Level.FINEST, "Metric changed but cannot map to instrument: {0}", metric.metricName());

From 8fce56b32f4b744a636bdd77415eb32f808139de Mon Sep 17 00:00:00 2001
From: Jack Berg 
Date: Fri, 1 Jul 2022 18:10:51 -0500
Subject: [PATCH 15/17] Adjust configuration pattern to not rely on
 GlobalOpenTelemetry

---
 .../OpenTelemetryMetricsReporterTest.java}    | 134 ++++++++++----
 .../kafkaclients/MetricsTest.java             |   4 +-
 .../OpenTelemetryKafkaMetrics.java            | 174 +++++-------------
 .../{ => internal}/InstrumentDescriptor.java  |   2 +-
 .../{ => internal}/KafkaMetricRegistry.java   |   6 +-
 .../OpenTelemetryMetricsReporter.java         | 159 ++++++++++++++++
 .../{ => internal}/RegisteredObservable.java  |   2 +-
 7 files changed, 310 insertions(+), 171 deletions(-)
 rename instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/{OpenTelemetryKafkaMetricsTest.java => internal/OpenTelemetryMetricsReporterTest.java} (81%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/{ => internal}/InstrumentDescriptor.java (93%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/{ => internal}/KafkaMetricRegistry.java (94%)
 create mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/{ => internal}/RegisteredObservable.java (92%)

diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java
similarity index 81%
rename from instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java
index 2d5676cadfef..19d0784de3df 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetricsTest.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients;
+package io.opentelemetry.instrumentation.kafkaclients.internal;
 
 import static java.lang.System.lineSeparator;
 import static java.util.Comparator.comparing;
@@ -12,9 +12,11 @@
 import static java.util.stream.Collectors.toList;
 import static java.util.stream.Collectors.toSet;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 import com.google.auto.value.AutoValue;
 import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.instrumentation.kafkaclients.OpenTelemetryKafkaMetrics;
 import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension;
 import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension;
 import io.opentelemetry.sdk.metrics.data.MetricData;
@@ -31,6 +33,7 @@
 import java.util.Optional;
 import java.util.Random;
 import java.util.Set;
+import org.apache.kafka.clients.CommonClientConfigs;
 import org.apache.kafka.clients.consumer.ConsumerConfig;
 import org.apache.kafka.clients.consumer.KafkaConsumer;
 import org.apache.kafka.clients.producer.KafkaProducer;
@@ -41,8 +44,9 @@
 import org.apache.kafka.common.metrics.MetricsReporter;
 import org.apache.kafka.common.serialization.ByteArrayDeserializer;
 import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.junit.jupiter.api.AfterAll;
 import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.RegisterExtension;
 import org.slf4j.Logger;
@@ -50,39 +54,55 @@
 import org.testcontainers.containers.KafkaContainer;
 import org.testcontainers.containers.output.Slf4jLogConsumer;
 import org.testcontainers.containers.wait.strategy.Wait;
-import org.testcontainers.junit.jupiter.Container;
 import org.testcontainers.junit.jupiter.Testcontainers;
 import org.testcontainers.utility.DockerImageName;
 
+/**
+ * This class is internal and is hence not for public use. Its APIs are unstable and can change at
+ * any time.
+ */
 @Testcontainers
-abstract class OpenTelemetryKafkaMetricsTest {
+public abstract class OpenTelemetryMetricsReporterTest {
 
-  private static final Logger logger = LoggerFactory.getLogger(OpenTelemetryKafkaMetricsTest.class);
+  private static final Logger logger =
+      LoggerFactory.getLogger(OpenTelemetryMetricsReporterTest.class);
 
   private static final List TOPICS = Arrays.asList("foo", "bar", "baz", "qux");
   private static final Random RANDOM = new Random();
 
-  @Container
-  KafkaContainer kafka =
-      new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3"))
-          .withLogConsumer(new Slf4jLogConsumer(logger))
-          .waitingFor(Wait.forLogMessage(".*started \\(kafka.server.KafkaServer\\).*", 1))
-          .withStartupTimeout(Duration.ofMinutes(1));
-
   @RegisterExtension
   static final InstrumentationExtension testing = LibraryInstrumentationExtension.create();
 
-  private KafkaProducer producer;
-  private KafkaConsumer consumer;
+  private static KafkaContainer kafka;
+  private static KafkaProducer producer;
+  private static KafkaConsumer consumer;
+
+  @BeforeAll
+  static void beforeAll() {
+    kafka =
+        new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3"))
+            .withLogConsumer(new Slf4jLogConsumer(logger))
+            .waitingFor(Wait.forLogMessage(".*started \\(kafka.server.KafkaServer\\).*", 1))
+            .withStartupTimeout(Duration.ofMinutes(1));
+    kafka.start();
+    producer = new KafkaProducer<>(producerConfig());
+    consumer = new KafkaConsumer<>(consumerConfig());
+  }
 
-  @BeforeEach
-  void setup() {
-    String metricReporters =
-        OpenTelemetryKafkaMetrics.class.getName() + "," + TestMetricsReporter.class.getName();
+  @AfterAll
+  static void afterAll() {
+    kafka.stop();
+    producer.close();
+    consumer.close();
+  }
+
+  @AfterEach
+  void tearDown() {
+    OpenTelemetryMetricsReporter.resetForTest();
+  }
 
+  private static Map producerConfig() {
     Map producerConfig = new HashMap<>();
-    // Register OpenTelemetryKafkaMetrics as reporter
-    producerConfig.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, metricReporters);
     producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
     producerConfig.put(ProducerConfig.CLIENT_ID_CONFIG, "sample-client-id");
     producerConfig.put(
@@ -90,11 +110,13 @@ void setup() {
     producerConfig.put(
         ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
     producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
-    producer = new KafkaProducer<>(producerConfig);
+    producerConfig.putAll(
+        OpenTelemetryKafkaMetrics.getConfigProperties(testing.getOpenTelemetry()));
+    return producerConfig;
+  }
 
+  private static Map consumerConfig() {
     Map consumerConfig = new HashMap<>();
-    // Register OpenTelemetryKafkaMetrics as reporter
-    consumerConfig.put(ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG, metricReporters);
     consumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
     consumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-group");
     consumerConfig.put(
@@ -102,14 +124,56 @@ void setup() {
     consumerConfig.put(
         ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
     consumerConfig.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2);
-    consumer = new KafkaConsumer<>(consumerConfig);
+    consumerConfig.putAll(
+        OpenTelemetryKafkaMetrics.getConfigProperties(testing.getOpenTelemetry()));
+    consumerConfig.merge(
+        CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
+        TestMetricsReporter.class.getName(),
+        (o, o2) -> o + "," + o2);
+    return consumerConfig;
   }
 
-  @AfterEach
-  void tearDown() {
-    OpenTelemetryKafkaMetrics.resetForTest();
-    producer.close();
-    consumer.close();
+  @Test
+  void badConfig() {
+    // Bad producer config
+    assertThatThrownBy(
+            () -> {
+              Map producerConfig = producerConfig();
+              producerConfig.remove(OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE);
+              new KafkaProducer<>(producerConfig).close();
+            })
+        .hasRootCauseInstanceOf(IllegalStateException.class)
+        .hasRootCauseMessage("Missing required configuration property: opentelemetry.instance");
+    assertThatThrownBy(
+            () -> {
+              Map producerConfig = producerConfig();
+              producerConfig.put(
+                  OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE, "foo");
+              new KafkaProducer<>(producerConfig).close();
+            })
+        .hasRootCauseInstanceOf(IllegalStateException.class)
+        .hasRootCauseMessage(
+            "Configuration property opentelemetry.instance is not instance of OpenTelemetry");
+
+    // Bad consumer config
+    assertThatThrownBy(
+            () -> {
+              Map consumerConfig = consumerConfig();
+              consumerConfig.remove(OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE);
+              new KafkaConsumer<>(consumerConfig).close();
+            })
+        .hasRootCauseInstanceOf(IllegalStateException.class)
+        .hasRootCauseMessage("Missing required configuration property: opentelemetry.instance");
+    assertThatThrownBy(
+            () -> {
+              Map consumerConfig = consumerConfig();
+              consumerConfig.put(
+                  OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE, "foo");
+              new KafkaConsumer<>(consumerConfig).close();
+            })
+        .hasRootCauseInstanceOf(IllegalStateException.class)
+        .hasRootCauseMessage(
+            "Configuration property opentelemetry.instance is not instance of OpenTelemetry");
   }
 
   @Test
@@ -338,7 +402,7 @@ void observeMetrics() {
     printMappingTable();
   }
 
-  void produceRecords() {
+  private static void produceRecords() {
     for (int i = 0; i < 100; i++) {
       producer.send(
           new ProducerRecord<>(
@@ -350,7 +414,7 @@ void produceRecords() {
     }
   }
 
-  void consumeRecords() {
+  private static void consumeRecords() {
     consumer.subscribe(TOPICS);
     Instant stopTime = Instant.now().plusSeconds(10);
     while (Instant.now().isBefore(stopTime)) {
@@ -373,7 +437,7 @@ private static void printMappingTable() {
     Map> kafkaMetricsByGroup =
         TestMetricsReporter.seenMetrics.stream().collect(groupingBy(KafkaMetricId::getGroup));
     List registeredObservables =
-        OpenTelemetryKafkaMetrics.getRegisteredObservables();
+        OpenTelemetryMetricsReporter.getRegisteredObservables();
     // Iterate through groups in alpha order
     for (String group : kafkaMetricsByGroup.keySet().stream().sorted().collect(toList())) {
       List kafkaMetricIds =
@@ -410,6 +474,10 @@ private static void printMappingTable() {
     logger.info("Mapping table" + System.lineSeparator() + sb);
   }
 
+  /**
+   * This class is internal and is hence not for public use. Its APIs are unstable and can change at
+   * any time.
+   */
   public static class TestMetricsReporter implements MetricsReporter {
 
     private static final Set seenMetrics = new HashSet<>();
@@ -444,7 +512,7 @@ abstract static class KafkaMetricId {
     abstract Set getAttributeKeys();
 
     static KafkaMetricId create(MetricName metricName) {
-      return new AutoValue_OpenTelemetryKafkaMetricsTest_KafkaMetricId(
+      return new AutoValue_OpenTelemetryMetricsReporterTest_KafkaMetricId(
           metricName.group(), metricName.name(), metricName.tags().keySet());
     }
   }
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
index 0c3bc151f4e3..78fe35317979 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
@@ -5,4 +5,6 @@
 
 package io.opentelemetry.instrumentation.kafkaclients;
 
-public class MetricsTest extends OpenTelemetryKafkaMetricsTest {}
+import io.opentelemetry.instrumentation.kafkaclients.internal.OpenTelemetryMetricsReporterTest;
+
+public class MetricsTest extends OpenTelemetryMetricsReporterTest {}
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
index 857ab52d681e..2fa2088f50b3 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
@@ -5,148 +5,58 @@
 
 package io.opentelemetry.instrumentation.kafkaclients;
 
-import io.opentelemetry.api.GlobalOpenTelemetry;
 import io.opentelemetry.api.OpenTelemetry;
-import io.opentelemetry.api.common.AttributeKey;
-import io.opentelemetry.api.metrics.Meter;
-import io.opentelemetry.instrumentation.api.internal.GuardedBy;
-import java.util.ArrayList;
-import java.util.Iterator;
-import java.util.List;
+import io.opentelemetry.instrumentation.kafkaclients.internal.OpenTelemetryMetricsReporter;
+import java.util.Collections;
+import java.util.HashMap;
 import java.util.Map;
-import java.util.Set;
-import java.util.logging.Level;
-import java.util.logging.Logger;
-import org.apache.kafka.common.metrics.KafkaMetric;
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.KafkaProducer;
 import org.apache.kafka.common.metrics.MetricsReporter;
 
 /**
- * A {@link MetricsReporter} which bridges Kafka metrics to OpenTelemetry metrics.
+ * Record kafka client metrics to OpenTelemetry.
  *
- * 
To use, configure {@link GlobalOpenTelemetry} instance via {@link
- * GlobalOpenTelemetry#set(OpenTelemetry)}, and include a reference to this class in kafka producer
- * or consumer configuration, i.e.:
- *
- * 
{@code
- * //    Map config = new HashMap<>();
- * //    // Register OpenTelemetryKafkaMetrics as reporter
- * //    config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName());
- * //    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
- * //    ...
- * //    try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
- * }

+ * @see #getConfigProperties(OpenTelemetry)
  */
-public class OpenTelemetryKafkaMetrics implements MetricsReporter {
-
-  private static final Logger logger = Logger.getLogger(OpenTelemetryKafkaMetrics.class.getName());
-  private static volatile Meter meter;
-
-  private static final Object lock = new Object();
-
-  @GuardedBy("lock")
-  private static final List registeredObservables = new ArrayList<>();
+public class OpenTelemetryKafkaMetrics {
 
   /**
-   * Reset for test by reseting the {@link #meter} to {@code null} and closing all registered
-   * instruments.
+   * Produces a set of kafka client config properties (consumer or producer) to register a {@link
+   * MetricsReporter} that records metrics to the {@code openTelemetry} instance. Add these
+   * resulting properties to the configuration map used to initialize a {@link KafkaConsumer} or
+   * {@link KafkaProducer}.
+   *
+   * 
For producers:
+   *
+   * 
{@code
+   * //    Map config = new HashMap<>();
+   * //    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
+   * //    config.putAll(OpenTelemetryKafka.openTelemetryConfigProperties(openTelemetry);
+   * //    try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
+   * }

+   *
+   * 
For consumers:
+   *
+   * 
{@code
+   * //    Map config = new HashMap<>();
+   * //    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
+   * //    config.putAll(OpenTelemetryKafka.openTelemetryConfigProperties(openTelemetry);
+   * //    try (KafkaConsumer consumer = new KafkaConsumer<>(config)) { ... }
+   * }

+   *
+   * @param openTelemetry the {@link OpenTelemetry} used to record metrics
+   * @return the kafka client properties
    */
-  static void resetForTest() {
-    meter = null;
-    closeAllInstruments();
-  }
-
-  // Visible for test
-  static List getRegisteredObservables() {
-    synchronized (lock) {
-      return new ArrayList<>(registeredObservables);
-    }
-  }
-
-  @Override
-  public void init(List metrics) {
-    metrics.forEach(this::metricChange);
-  }
-
-  @Override
-  public void metricChange(KafkaMetric metric) {
-    // Maybe initialize meter
-    Meter currentMeter = meter;
-    if (currentMeter == null) {
-      currentMeter = meter = GlobalOpenTelemetry.getMeter("io.opentelemetry.kafka-clients");
-    }
-
-    RegisteredObservable registeredObservable =
-        KafkaMetricRegistry.getRegisteredObservable(currentMeter, metric);
-    if (registeredObservable == null) {
-      logger.log(
-          Level.FINEST, "Metric changed but cannot map to instrument: {0}", metric.metricName());
-      return;
-    }
-
-    Set> attributeKeys = registeredObservable.getAttributes().asMap().keySet();
-    synchronized (lock) {
-      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
-        RegisteredObservable curRegisteredObservable = it.next();
-        Set> curAttributeKeys =
-            curRegisteredObservable.getAttributes().asMap().keySet();
-        if (curRegisteredObservable.getKafkaMetricName().equals(metric.metricName())) {
-          logger.log(Level.FINEST, "Replacing instrument: {0}", curRegisteredObservable);
-          closeInstrument(curRegisteredObservable.getObservable());
-          it.remove();
-        } else if (curRegisteredObservable
-                .getInstrumentDescriptor()
-                .equals(registeredObservable.getInstrumentDescriptor())
-            && attributeKeys.size() > curAttributeKeys.size()
-            && attributeKeys.containsAll(curAttributeKeys)) {
-          logger.log(
-              Level.FINEST,
-              "Replacing instrument with higher dimension version: {0}",
-              curRegisteredObservable);
-          closeInstrument(curRegisteredObservable.getObservable());
-          it.remove();
-        }
-      }
-
-      registeredObservables.add(registeredObservable);
-    }
-  }
-
-  @Override
-  public void metricRemoval(KafkaMetric metric) {
-    logger.log(Level.FINEST, "Metric removed: {0}", metric.metricName());
-    synchronized (lock) {
-      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
-        RegisteredObservable current = it.next();
-        if (current.getKafkaMetricName().equals(metric.metricName())) {
-          closeInstrument(current.getObservable());
-          it.remove();
-        }
-      }
-    }
-  }
-
-  @Override
-  public void close() {
-    closeAllInstruments();
-  }
-
-  static void closeAllInstruments() {
-    synchronized (lock) {
-      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
-        closeInstrument(it.next().getObservable());
-        it.remove();
-      }
-    }
-  }
-
-  private static void closeInstrument(AutoCloseable observable) {
-    try {
-      observable.close();
-    } catch (Exception e) {
-      throw new IllegalStateException("Error occurred closing instrument", e);
-    }
+  public static Map getConfigProperties(OpenTelemetry openTelemetry) {
+    Map config = new HashMap<>();
+    config.put(
+        CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
+        OpenTelemetryMetricsReporter.class.getName());
+    config.put(OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE, openTelemetry);
+    return Collections.unmodifiableMap(config);
   }
 
-  @Override
-  public void configure(Map configs) {}
+  private OpenTelemetryKafkaMetrics() {}
 }
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java
similarity index 93%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java
index 61a16dd222a1..01fcc03feeb5 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/InstrumentDescriptor.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients;
+package io.opentelemetry.instrumentation.kafkaclients.internal;
 
 import com.google.auto.value.AutoValue;
 
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java
similarity index 94%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java
index 97ac761e171c..73f7078873c1 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaMetricRegistry.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java
@@ -3,10 +3,10 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients;
+package io.opentelemetry.instrumentation.kafkaclients.internal;
 
-import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER;
-import static io.opentelemetry.instrumentation.kafkaclients.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE;
+import static io.opentelemetry.instrumentation.kafkaclients.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER;
+import static io.opentelemetry.instrumentation.kafkaclients.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE;
 
 import io.opentelemetry.api.common.Attributes;
 import io.opentelemetry.api.common.AttributesBuilder;
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java
new file mode 100644
index 000000000000..633670bb31ff
--- /dev/null
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java
@@ -0,0 +1,159 @@
+/*
+ * Copyright The OpenTelemetry Authors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package io.opentelemetry.instrumentation.kafkaclients.internal;
+
+import io.opentelemetry.api.OpenTelemetry;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.metrics.Meter;
+import io.opentelemetry.instrumentation.api.internal.GuardedBy;
+import io.opentelemetry.instrumentation.kafkaclients.OpenTelemetryKafkaMetrics;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.logging.Level;
+import java.util.logging.Logger;
+import org.apache.kafka.common.metrics.KafkaMetric;
+import org.apache.kafka.common.metrics.MetricsReporter;
+
+/**
+ * A {@link MetricsReporter} which bridges Kafka metrics to OpenTelemetry metrics.
+ *
+ * 
To configure, use {@link OpenTelemetryKafkaMetrics#getConfigProperties(OpenTelemetry)}.
+ *
+ * 
This class is internal and is hence not for public use. Its APIs are unstable and can change
+ * at any time.
+ */
+public class OpenTelemetryMetricsReporter implements MetricsReporter {
+
+  public static final String CONFIG_KEY_OPENTELEMETRY_INSTANCE = "opentelemetry.instance";
+
+  private static final Logger logger =
+      Logger.getLogger(OpenTelemetryMetricsReporter.class.getName());
+  private volatile Meter meter;
+
+  private static final Object lock = new Object();
+
+  @GuardedBy("lock")
+  private static final List registeredObservables = new ArrayList<>();
+
+  /**
+   * Reset for test by reseting the {@link #meter} to {@code null} and closing all registered
+   * instruments.
+   */
+  static void resetForTest() {
+    closeAllInstruments();
+  }
+
+  // Visible for test
+  static List getRegisteredObservables() {
+    synchronized (lock) {
+      return new ArrayList<>(registeredObservables);
+    }
+  }
+
+  @Override
+  public void init(List metrics) {
+    metrics.forEach(this::metricChange);
+  }
+
+  @Override
+  public void metricChange(KafkaMetric metric) {
+    Meter currentMeter = meter;
+    if (currentMeter == null) {
+      // Ignore if meter hasn't been initialized in configure(Map> attributeKeys = registeredObservable.getAttributes().asMap().keySet();
+    synchronized (lock) {
+      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
+        RegisteredObservable curRegisteredObservable = it.next();
+        Set> curAttributeKeys =
+            curRegisteredObservable.getAttributes().asMap().keySet();
+        if (curRegisteredObservable.getKafkaMetricName().equals(metric.metricName())) {
+          logger.log(Level.FINEST, "Replacing instrument: {0}", curRegisteredObservable);
+          closeInstrument(curRegisteredObservable.getObservable());
+          it.remove();
+        } else if (curRegisteredObservable
+                .getInstrumentDescriptor()
+                .equals(registeredObservable.getInstrumentDescriptor())
+            && attributeKeys.size() > curAttributeKeys.size()
+            && attributeKeys.containsAll(curAttributeKeys)) {
+          logger.log(
+              Level.FINEST,
+              "Replacing instrument with higher dimension version: {0}",
+              curRegisteredObservable);
+          closeInstrument(curRegisteredObservable.getObservable());
+          it.remove();
+        }
+      }
+
+      registeredObservables.add(registeredObservable);
+    }
+  }
+
+  @Override
+  public void metricRemoval(KafkaMetric metric) {
+    logger.log(Level.FINEST, "Metric removed: {0}", metric.metricName());
+    synchronized (lock) {
+      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
+        RegisteredObservable current = it.next();
+        if (current.getKafkaMetricName().equals(metric.metricName())) {
+          closeInstrument(current.getObservable());
+          it.remove();
+        }
+      }
+    }
+  }
+
+  @Override
+  public void close() {
+    closeAllInstruments();
+  }
+
+  private static void closeAllInstruments() {
+    synchronized (lock) {
+      for (Iterator it = registeredObservables.iterator(); it.hasNext(); ) {
+        closeInstrument(it.next().getObservable());
+        it.remove();
+      }
+    }
+  }
+
+  private static void closeInstrument(AutoCloseable observable) {
+    try {
+      observable.close();
+    } catch (Exception e) {
+      throw new IllegalStateException("Error occurred closing instrument", e);
+    }
+  }
+
+  @Override
+  public void configure(Map configs) {
+    Object openTelemetry = configs.get(CONFIG_KEY_OPENTELEMETRY_INSTANCE);
+    if (openTelemetry == null) {
+      throw new IllegalStateException(
+          "Missing required configuration property: " + CONFIG_KEY_OPENTELEMETRY_INSTANCE);
+    }
+    if (!(openTelemetry instanceof OpenTelemetry)) {
+      throw new IllegalStateException(
+          "Configuration property "
+              + CONFIG_KEY_OPENTELEMETRY_INSTANCE
+              + " is not instance of OpenTelemetry");
+    }
+    meter = ((OpenTelemetry) openTelemetry).getMeter("io.opentelemetry.kafka-clients");
+  }
+}
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java
similarity index 92%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java
index 0ae8ded63f62..71a58d073fe4 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/RegisteredObservable.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients;
+package io.opentelemetry.instrumentation.kafkaclients.internal;
 
 import com.google.auto.value.AutoValue;
 import io.opentelemetry.api.common.Attributes;

From b367d2d3fe75cb531593a4c1cde2975b41f66ecd Mon Sep 17 00:00:00 2001
From: Jack Berg 
Date: Mon, 4 Jul 2022 22:55:14 -0500
Subject: [PATCH 16/17] Merge into KafkaTelemetry

---
 .../testing/build.gradle.kts                  |  4 --
 .../library/build.gradle.kts                  |  4 ++
 .../kafkaclients/KafkaTelemetry.java          | 43 +++++++++++++
 .../kafkaclients/MetricsTest.java             | 10 ---
 .../OpenTelemetryMetricsReporterTest.java     | 20 +++---
 .../kafka-clients-common/library/README.md    | 18 ++----
 .../internal/InstrumentDescriptor.java        |  2 +-
 .../internal/KafkaMetricRegistry.java         |  8 +--
 .../OpenTelemetryMetricsReporter.java         | 11 ++--
 .../internal/RegisteredObservable.java        |  2 +-
 .../OpenTelemetryKafkaMetrics.java            | 62 -------------------
 11 files changed, 75 insertions(+), 109 deletions(-)
 delete mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
 rename instrumentation/kafka/kafka-clients/{kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients => kafka-clients-2.6/library/src/test/java/io/opentelemetry/instrumentation/kafka}/internal/OpenTelemetryMetricsReporterTest.java (97%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/{kafkaclients => kafka}/internal/InstrumentDescriptor.java (93%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/{kafkaclients => kafka}/internal/KafkaMetricRegistry.java (94%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/{kafkaclients => kafka}/internal/OpenTelemetryMetricsReporter.java (94%)
 rename instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/{kafkaclients => kafka}/internal/RegisteredObservable.java (92%)
 delete mode 100644 instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java

diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts
index 3058fd208b2f..b8dc4adc702b 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/build.gradle.kts
@@ -10,8 +10,4 @@ dependencies {
   implementation(project(":instrumentation:kafka:kafka-clients:kafka-clients-common:library"))
 
   implementation("org.testcontainers:kafka")
-  implementation("org.testcontainers:junit-jupiter")
-
-  compileOnly("com.google.auto.value:auto-value-annotations")
-  annotationProcessor("com.google.auto.value:auto-value")
 }
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/build.gradle.kts b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/build.gradle.kts
index beece7127590..87ab9440f76d 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/build.gradle.kts
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/build.gradle.kts
@@ -10,6 +10,10 @@ dependencies {
   testImplementation("com.fasterxml.jackson.core:jackson-databind:2.10.2")
 
   testImplementation("org.testcontainers:kafka")
+  testImplementation("org.testcontainers:junit-jupiter")
+
+  testCompileOnly("com.google.auto.value:auto-value-annotations")
+  testAnnotationProcessor("com.google.auto.value:auto-value")
 }
 
 tasks {
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaTelemetry.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaTelemetry.java
index 1854d885f03d..62cf703a80aa 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaTelemetry.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/KafkaTelemetry.java
@@ -17,17 +17,25 @@
 import io.opentelemetry.instrumentation.api.instrumenter.Instrumenter;
 import io.opentelemetry.instrumentation.kafka.internal.KafkaConsumerRecordGetter;
 import io.opentelemetry.instrumentation.kafka.internal.KafkaHeadersSetter;
+import io.opentelemetry.instrumentation.kafka.internal.OpenTelemetryMetricsReporter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
 import java.util.concurrent.Future;
 import java.util.function.BiFunction;
 import java.util.logging.Logger;
+import org.apache.kafka.clients.CommonClientConfigs;
 import org.apache.kafka.clients.consumer.Consumer;
 import org.apache.kafka.clients.consumer.ConsumerRecord;
 import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
 import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.KafkaProducer;
 import org.apache.kafka.clients.producer.Producer;
 import org.apache.kafka.clients.producer.ProducerRecord;
 import org.apache.kafka.clients.producer.RecordMetadata;
 import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.metrics.MetricsReporter;
 
 public final class KafkaTelemetry {
   private static final Logger logger = Logger.getLogger(KafkaTelemetry.class.getName());
@@ -76,6 +84,41 @@ public  Consumer wrap(Consumer consumer) {
     return new TracingConsumer<>(consumer, this);
   }
 
+  /**
+   * Produces a set of kafka client config properties (consumer or producer) to register a {@link
+   * MetricsReporter} that records metrics to an {@code openTelemetry} instance. Add these resulting
+   * properties to the configuration map used to initialize a {@link KafkaConsumer} or {@link
+   * KafkaProducer}.
+   *
+   * 
For producers:
+   *
+   * 
{@code
+   * //    Map config = new HashMap<>();
+   * //    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
+   * //    config.putAll(kafkaTelemetry.metricConfigProperties());
+   * //    try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
+   * }

+   *
+   * 
For consumers:
+   *
+   * 
{@code
+   * //    Map config = new HashMap<>();
+   * //    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
+   * //    config.putAll(kafkaTelemetry.metricConfigProperties());
+   * //    try (KafkaConsumer consumer = new KafkaConsumer<>(config)) { ... }
+   * }

+   *
+   * @return the kafka client properties
+   */
+  public Map metricConfigProperties() {
+    Map config = new HashMap<>();
+    config.put(
+        CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
+        OpenTelemetryMetricsReporter.class.getName());
+    config.put(OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE, openTelemetry);
+    return Collections.unmodifiableMap(config);
+  }
+
   /**
    * Build and inject span into record.
    *
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
deleted file mode 100644
index 78fe35317979..000000000000
--- a/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/groovy/io/opentelemetry/instrumentation/kafkaclients/MetricsTest.java
+++ /dev/null
@@ -1,10 +0,0 @@
-/*
- * Copyright The OpenTelemetry Authors
- * SPDX-License-Identifier: Apache-2.0
- */
-
-package io.opentelemetry.instrumentation.kafkaclients;
-
-import io.opentelemetry.instrumentation.kafkaclients.internal.OpenTelemetryMetricsReporterTest;
-
-public class MetricsTest extends OpenTelemetryMetricsReporterTest {}
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporterTest.java
similarity index 97%
rename from instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporterTest.java
index 19d0784de3df..ddf5bf6f3b13 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-0.11/testing/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporterTest.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/test/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporterTest.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients.internal;
+package io.opentelemetry.instrumentation.kafka.internal;
 
 import static java.lang.System.lineSeparator;
 import static java.util.Comparator.comparing;
@@ -16,7 +16,7 @@
 
 import com.google.auto.value.AutoValue;
 import io.opentelemetry.api.common.AttributeKey;
-import io.opentelemetry.instrumentation.kafkaclients.OpenTelemetryKafkaMetrics;
+import io.opentelemetry.instrumentation.kafkaclients.KafkaTelemetry;
 import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension;
 import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension;
 import io.opentelemetry.sdk.metrics.data.MetricData;
@@ -57,12 +57,8 @@
 import org.testcontainers.junit.jupiter.Testcontainers;
 import org.testcontainers.utility.DockerImageName;
 
-/**
- * This class is internal and is hence not for public use. Its APIs are unstable and can change at
- * any time.
- */
 @Testcontainers
-public abstract class OpenTelemetryMetricsReporterTest {
+class OpenTelemetryMetricsReporterTest {
 
   private static final Logger logger =
       LoggerFactory.getLogger(OpenTelemetryMetricsReporterTest.class);
@@ -111,7 +107,11 @@ private static Map producerConfig() {
         ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
     producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
     producerConfig.putAll(
-        OpenTelemetryKafkaMetrics.getConfigProperties(testing.getOpenTelemetry()));
+        KafkaTelemetry.create(testing.getOpenTelemetry()).metricConfigProperties());
+    producerConfig.merge(
+        CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
+        TestMetricsReporter.class.getName(),
+        (o, o2) -> o + "," + o2);
     return producerConfig;
   }
 
@@ -125,7 +125,7 @@ private static Map consumerConfig() {
         ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
     consumerConfig.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2);
     consumerConfig.putAll(
-        OpenTelemetryKafkaMetrics.getConfigProperties(testing.getOpenTelemetry()));
+        KafkaTelemetry.create(testing.getOpenTelemetry()).metricConfigProperties());
     consumerConfig.merge(
         CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
         TestMetricsReporter.class.getName(),
@@ -418,7 +418,7 @@ private static void consumeRecords() {
     consumer.subscribe(TOPICS);
     Instant stopTime = Instant.now().plusSeconds(10);
     while (Instant.now().isBefore(stopTime)) {
-      consumer.poll(1000);
+      consumer.poll(Duration.ofSeconds(1));
     }
   }
 
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md
index ed2610e58a1e..eba57bd401f0 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md
@@ -3,23 +3,15 @@
 The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface.
 OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry.
 
-To use, configure `GlobalOpenTelemetry` with an OpenTelemetry instance
-via `GlobalOpenTelemetry#set(OpenTelemetry)`, and include a reference to this
-class in kafka producer or consumer configuration, i.e.:
-
-```java
-Map config = new HashMap<>();
-config.put(ProducerConfig.METRIC_REPORTER_CLASSES_CONFIG, OpenTelemetryKafkaMetrics.class.getName());
-config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getKafkaConnectString());
-...
-try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
-```
+To use, add merge the config properties
+from `KafkaTelemetry.create(OpenTelemetry).metricConfigProperties()`
+with the configuration used when creating your producer or consumer.
 
 Note: Kafka reports several metrics at multiple attribute granularities. For
 example, `records-consumed-total` is reported with attribute key `[client-id]`
 and `[client-id, topic]`. If you analyze the sum of records consumed, ignoring dimensions, backends
-are likely to double count. To alleviate this, `OpenTelemetryKafkaMetrics` detects this
-scenario and only records the most granular set of attributes available. In the case
+are likely to double count. The implementation detects this scenario and only records the most
+granular set of attributes available. In the case
 of `records-consumed-total`, it reports `[client-id, topic]` and ignores `[client-id]`.
 
 The following table shows the full set of metrics exposed by the kafka client, and the corresponding
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/InstrumentDescriptor.java
similarity index 93%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/InstrumentDescriptor.java
index 01fcc03feeb5..51b52130b904 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/InstrumentDescriptor.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/InstrumentDescriptor.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients.internal;
+package io.opentelemetry.instrumentation.kafka.internal;
 
 import com.google.auto.value.AutoValue;
 
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/KafkaMetricRegistry.java
similarity index 94%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/KafkaMetricRegistry.java
index 73f7078873c1..d1beb57ffa4e 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/KafkaMetricRegistry.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/KafkaMetricRegistry.java
@@ -3,10 +3,10 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients.internal;
+package io.opentelemetry.instrumentation.kafka.internal;
 
-import static io.opentelemetry.instrumentation.kafkaclients.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER;
-import static io.opentelemetry.instrumentation.kafkaclients.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE;
+import static io.opentelemetry.instrumentation.kafka.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_COUNTER;
+import static io.opentelemetry.instrumentation.kafka.internal.InstrumentDescriptor.INSTRUMENT_TYPE_DOUBLE_OBSERVABLE_GAUGE;
 
 import io.opentelemetry.api.common.Attributes;
 import io.opentelemetry.api.common.AttributesBuilder;
@@ -26,7 +26,7 @@
 import org.apache.kafka.common.metrics.Measurable;
 
 /** A registry mapping kafka metrics to corresponding OpenTelemetry metric definitions. */
-class KafkaMetricRegistry {
+final class KafkaMetricRegistry {
 
   private static final Set groups = new HashSet<>(Arrays.asList("consumer", "producer"));
   private static final Map, String> measureableToInstrumentType = new HashMap<>();
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporter.java
similarity index 94%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporter.java
index 633670bb31ff..f888ff621613 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/OpenTelemetryMetricsReporter.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/OpenTelemetryMetricsReporter.java
@@ -3,13 +3,12 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients.internal;
+package io.opentelemetry.instrumentation.kafka.internal;
 
 import io.opentelemetry.api.OpenTelemetry;
 import io.opentelemetry.api.common.AttributeKey;
 import io.opentelemetry.api.metrics.Meter;
 import io.opentelemetry.instrumentation.api.internal.GuardedBy;
-import io.opentelemetry.instrumentation.kafkaclients.OpenTelemetryKafkaMetrics;
 import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.List;
@@ -23,12 +22,16 @@
 /**
  * A {@link MetricsReporter} which bridges Kafka metrics to OpenTelemetry metrics.
  *
- * 
To configure, use {@link OpenTelemetryKafkaMetrics#getConfigProperties(OpenTelemetry)}.
+ * 
To configure, use:
+ *
+ * 
<{@code
+ * // KafkaTelemetry.KafkaTelemetry.create(OpenTelemetry).metricConfigProperties()
+ * }

  *
  * 
This class is internal and is hence not for public use. Its APIs are unstable and can change
  * at any time.
  */
-public class OpenTelemetryMetricsReporter implements MetricsReporter {
+public final class OpenTelemetryMetricsReporter implements MetricsReporter {
 
   public static final String CONFIG_KEY_OPENTELEMETRY_INSTANCE = "opentelemetry.instance";
 
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/RegisteredObservable.java
similarity index 92%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java
rename to instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/RegisteredObservable.java
index 71a58d073fe4..4717b63d53f6 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/internal/RegisteredObservable.java
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafka/internal/RegisteredObservable.java
@@ -3,7 +3,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-package io.opentelemetry.instrumentation.kafkaclients.internal;
+package io.opentelemetry.instrumentation.kafka.internal;
 
 import com.google.auto.value.AutoValue;
 import io.opentelemetry.api.common.Attributes;
diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java b/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
deleted file mode 100644
index 2fa2088f50b3..000000000000
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/OpenTelemetryKafkaMetrics.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Copyright The OpenTelemetry Authors
- * SPDX-License-Identifier: Apache-2.0
- */
-
-package io.opentelemetry.instrumentation.kafkaclients;
-
-import io.opentelemetry.api.OpenTelemetry;
-import io.opentelemetry.instrumentation.kafkaclients.internal.OpenTelemetryMetricsReporter;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.kafka.clients.CommonClientConfigs;
-import org.apache.kafka.clients.consumer.KafkaConsumer;
-import org.apache.kafka.clients.producer.KafkaProducer;
-import org.apache.kafka.common.metrics.MetricsReporter;
-
-/**
- * Record kafka client metrics to OpenTelemetry.
- *
- * @see #getConfigProperties(OpenTelemetry)
- */
-public class OpenTelemetryKafkaMetrics {
-
-  /**
-   * Produces a set of kafka client config properties (consumer or producer) to register a {@link
-   * MetricsReporter} that records metrics to the {@code openTelemetry} instance. Add these
-   * resulting properties to the configuration map used to initialize a {@link KafkaConsumer} or
-   * {@link KafkaProducer}.
-   *
-   * 
For producers:
-   *
-   * 
{@code
-   * //    Map config = new HashMap<>();
-   * //    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
-   * //    config.putAll(OpenTelemetryKafka.openTelemetryConfigProperties(openTelemetry);
-   * //    try (KafkaProducer producer = new KafkaProducer<>(config)) { ... }
-   * }

-   *
-   * 
For consumers:
-   *
-   * 
{@code
-   * //    Map config = new HashMap<>();
-   * //    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
-   * //    config.putAll(OpenTelemetryKafka.openTelemetryConfigProperties(openTelemetry);
-   * //    try (KafkaConsumer consumer = new KafkaConsumer<>(config)) { ... }
-   * }

-   *
-   * @param openTelemetry the {@link OpenTelemetry} used to record metrics
-   * @return the kafka client properties
-   */
-  public static Map getConfigProperties(OpenTelemetry openTelemetry) {
-    Map config = new HashMap<>();
-    config.put(
-        CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
-        OpenTelemetryMetricsReporter.class.getName());
-    config.put(OpenTelemetryMetricsReporter.CONFIG_KEY_OPENTELEMETRY_INSTANCE, openTelemetry);
-    return Collections.unmodifiableMap(config);
-  }
-
-  private OpenTelemetryKafkaMetrics() {}
-}

From 5f290f5519db20940fc37227d9e9d31c1ec12354 Mon Sep 17 00:00:00 2001
From: Jack Berg 
Date: Wed, 6 Jul 2022 10:28:05 -0500
Subject: [PATCH 17/17] Relocate readme and fix typo

---
 .../library/README.md                                           | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 rename instrumentation/kafka/kafka-clients/{kafka-clients-common => kafka-clients-2.6}/library/README.md (99%)

diff --git a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/README.md
similarity index 99%
rename from instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md
rename to instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/README.md
index eba57bd401f0..a7bf0e6cbde7 100644
--- a/instrumentation/kafka/kafka-clients/kafka-clients-common/library/README.md
+++ b/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/README.md
@@ -3,7 +3,7 @@
 The Kafka client exposes metrics via `org.apache.kafka.common.metrics.MetricsReporter` interface.
 OpenTelemetry provides an implementation that bridges the metrics into OpenTelemetry.
 
-To use, add merge the config properties
+To use, merge the config properties
 from `KafkaTelemetry.create(OpenTelemetry).metricConfigProperties()`
 with the configuration used when creating your producer or consumer.