Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

Latest commit

 

History

History
71 lines (47 loc) · 3.52 KB

inputs.md

File metadata and controls

71 lines (47 loc) · 3.52 KB
Raw

Inputs

All input options - except for the carbon input - use the metrics 2.0 format. See schema for more details.

How to send data to MT

see fakemetrics, tsdb-gw, carbon

Carbon

useful for traditional graphite plaintext protocol. Does not support pickle format.

Important: this input requires a carbon storage-schemas.conf file. Metrictank uses this file to determine the raw interval of the metrics, but it ignores all retention durations as well as intervals after the first, raw one since metrictank already has its own config mechanism for retention and aggregation.

note: it does not implement carbon2.0

Kafka-mdm (recommended)

This is the recommended input option if you want a queue. It also simplifies the operational model: since you can make nodes replay data you don't have to reassign primary/secondary roles at runtime, you can just restart write nodes and have them replay data, for example. Note that carbon-relay-ng can be used to pipe a carbon stream into Kafka.

The Kafka input supports 2 formats:

  • MetricData
  • MetricPoint

Both formats have a corresponding implementation in schema, making it trivial to implement your own producers (and consumers) if you use Golang.

MetricData

This format contains the data points as well as all metric identity data and metadata, in messagepack encoded messages (not JSON). This format is rather verbose and inefficient to encode/decode. See the MetricData documentation for format and implementation details.

MetricPoint

This format is a hand-written binary format that is much more compact and fast to encode/decode compared to MetricData. See the MetricPoint documentation for format and implementation details. It is a minimal format that only contains the series identifier, value and timestamp. As such, it is paramount that MetricPoint messages for each series have been preceded by a MetricData message for that series, so that metrictank has had the chance to add the metric information into its index. Otherwise metrictank will not recognize the ID and discard the point.

Note that the implementation has encode/decode function for the standard MetricPoint format, as well as a variant that does not encode the org-id part of the series id. For single-tenant environments, you can configure your producers and metrictank to not encode an org-id in all messages and rather just set it in configuration, this makes the message more compact, but won't work in multi-tenant environments.

MetricPointArray

This format supports multiple MetricPoint values in a single Kafka message. The message format is:

  1. a single prefixed format byte FormatMetricPointArray (0x05) followed by
  2. another byte for the MetricPoint sub-format, one of:
    • FormatMetricPoint (0x02)
    • FormatMetricPointWithoutOrg (0x03)
  3. a sequence of concatenated points (in the specified sub-format) with no delimiters.

Effective batching greatly reduces the message overhead from kafka (both for producers and Metrictank).

Future formats

In the future we plan to do more optimisations such as:

  • further compression (e.g. multiple points with shared timestamp).