Metrics query builder improvements

[srikanthccv]

• Add metric meta information to time_series_v3 and time_series_v2

  • description
  • unit
  • type
  • is_monotonic

• Update the current aggregate attribute API to include the metric meta information

• Add API endpoint for supported temporal and spatial aggregations for the metric type

• Add API endpoint for supported functions by category

• Update the current BuilderQuery payload to support the temporal and spatial aggregations for metrics datasource and functions for any data source

• Update the query builder logic to support the temporal and spatial aggregations for the metrics source

• Update the get time series result to support the functions

• Add support for functions that take the time series as input and return a time series as output with a function applied on each time series

• Add extensive tests for functions

• Update docs describing temporal and spatial aggregation better and add examples

• Migrate all known users and supported dashboard jsons on promql and CH query to the new BuilderQuery payload

More to be added after the PRD and design discussion.

[ankitnayan]

• Add multiple aggregations support in a single query? This will improve perf and save compute?

[srikanthccv]

Yes, it comes up often percentile queries. It applies to all signals and more generic to query builder #3156.

[srikanthccv]

@YounixM here are the points we discussed and the things that needed to be done from the client side

Functional Specification

• User should be asked to input the name of the metric

• User /autocomplete/aggregate_attributes to fetch the list of the matching metric names

  • The response from the API will have the type of the metric

• Based on the type of the metric, There are two possibilities

  1. Show both time and space aggregation options
  2. Show only space aggregation options

• The space aggregation from the UI side is a change from "Group By" text in the current builder to Sum By, Avg By, P99 By i.e. All space aggregation operators are shown in the dropdown.

• Any number of functions can be applied to a single query in any order

  • Each function has a name and option list of args

Following is the list of metric types and their expected behaviour

1. Sum (i.e Counter)

   • Show both time and space aggregation options
   • Options for time aggregation
     1. rate
     2. increase
   • Options for space aggregation
     1. sum
     2. avg
     3. min
     4. max

2. Gauge

   • Show both time and space aggregation options
   • Options for time aggregation
     1. latest
     2. sum
     3. avg
     4. min
     5. max
     6. count
     7. count_distinct
   • Options for space aggregation
     1. sum
     2. avg
     3. min
     4. max

3. Histogram

   • Show only space aggregation options
   • Options for space aggregation
     1. percentile_50
     2. percentile_75
     3. percentile_90
     4. percentile_95
     5. percentile_99

4. ExponentialHistogram

   • Show only space aggregation options
   • Options for space aggregation
     1. percentile_50
     2. percentile_75
     3. percentile_90
     4. percentile_95
     5. percentile_99

Following is the list of functions supported and the optional args expected

1. cutOffMin
   • Requires one threshold input from user
2. cutOffMax
   • Requires one threshold input from user
3. clampMin
   • Requires one threshold input from user
4. clampMax
   • Requires one threshold input from user
5. absolute
   • No input required
6. log2
   • No input required
7. log10
   • No input required
8. cumSum
   • No input required
9. ewma3
   • Requires one alpha input from user
10. ewma5
    • Requires one alpha input from user
11. ewma7
    • Requires one alpha input from user
12. median3
    • No input required
13. median5
    • No input required
14. median7
    • No input required
15. timeShift
    • Requires one shiftBy (in seconds) input from user

The list of new/updated fields expected in the payload

Note: Use v4/query_range for this new builder queries

1. timeAggregation - Selected time aggregation from the above list of options for each type

2. spaceAggregation - Selected space aggregation from the above list of options for each type

3. aggregateAttribute.type - Type of the selected metric (use the response from the /autocomplete/aggregate_attributes)

4. functions - List of functions and args

   • Example: {..., 'functions': [{'name': 'median5'}, {'name': 'timeShift', 'args': [1800]}], ...}

Non-functional requirements

1. Version wherever needed to make the dashboard remain backwards compatible and show the appropriate builder
2. Help users migrate by providing instructions and script
3. For the dashboard that can be migrated, migrate them with a script

[YounixM]

1. Sum
   Rate -Default Time aggregation
   Sum - Default Space aggregation

2. Guage
   Avg-Default Time aggregation
   Avg- Default Space aggregation

3. Histogram / Expo Histogram
   NA - Default Time aggregation
   P90 - Default Space aggregation

[ankitnayan]

@srikanthccv let's create a new issue for the items not done and close this

[srikanthccv]

All the user-facing tasks are done. Follow up in #4845.