Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mimir Query Engine: Add instant-vector functions #8256

Merged
merged 40 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
a297de4
Mimir query engine: Add support for histogram_count function
jhesketh May 31, 2024
6fd0e99
Support new limit pooling
jhesketh Jun 3, 2024
fb6b581
Test mixed data use cases
jhesketh Jun 3, 2024
0752b29
Refactor function calling
jhesketh Jun 3, 2024
3aa7014
Add support for histogram_sum
jhesketh Jun 3, 2024
5b02813
Add in simple function
jhesketh Jun 3, 2024
0fbd7b4
Update CHANGELOG
jhesketh Jun 3, 2024
0565a58
Fix lint
jhesketh Jun 3, 2024
c2e4ae5
Merge branch 'main' into jhesketh/promql
jhesketh Jun 4, 2024
6924f7c
Refactor functions into their own package
jhesketh Jun 4, 2024
c531127
Fix lint
jhesketh Jun 4, 2024
d41ac2d
Remove dropMetricName helper
jhesketh Jun 4, 2024
715dab0
Put series back into pool after count+sum
jhesketh Jun 4, 2024
0caf6e2
Rename function over types
jhesketh Jun 4, 2024
abc09ba
Add support for clamp function
jhesketh Jun 4, 2024
cf98d31
Move clamp into functions package
jhesketh Jun 4, 2024
13afa3f
We support scalars now
jhesketh Jun 4, 2024
2cef2c6
fix lint
jhesketh Jun 4, 2024
b9e9710
Re-arrange functions slightly
jhesketh Jun 5, 2024
88e0705
Rename methods+vars for consistency
jhesketh Jun 5, 2024
076a367
Fix native_histogram tests
jhesketh Jun 5, 2024
2ad5cbe
Ignore histogram with float functions
jhesketh Jun 5, 2024
827b2f7
Add tests for functions
jhesketh Jun 5, 2024
8024f1d
Remove scalar operator for now (along with clamp function)
jhesketh Jun 6, 2024
3c7c56b
Address review feedback
jhesketh Jun 6, 2024
d67f81f
Merge branch 'main' into jhesketh/promql
jhesketh Jun 6, 2024
30a3197
Rename instantVectorFunctionOperatorFactories to instantVectorFunctio…
jhesketh Jun 6, 2024
7da3fa5
Allow registering instantVectorFunctions
jhesketh Jun 6, 2024
881b816
Address review feedback
jhesketh Jun 12, 2024
bfc40d2
Add TestRegisterInstantVectorFunctionOperator
jhesketh Jun 12, 2024
cf89455
Merge branch 'main' into jhesketh/promql
jhesketh Jun 12, 2024
8f40b7d
Fix unit tests
jhesketh Jun 12, 2024
75506d6
Make clearer what are factories
jhesketh Jun 12, 2024
70791a7
Fix lint
jhesketh Jun 12, 2024
361070c
Test extra case
jhesketh Jun 12, 2024
2199afe
Enable trig functions
jhesketh Jun 12, 2024
f484280
Remove redundant tests
jhesketh Jun 12, 2024
5ff8972
Fix doc strings
jhesketh Jun 12, 2024
7b7f169
Make sure tests tidy themselves up
jhesketh Jun 12, 2024
b4f3234
Add check for NaN in acos
jhesketh Jun 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions pkg/streamingpromql/functions.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,19 @@ import (
"github.com/grafana/mimir/pkg/streamingpromql/types"
)

type InstantVectorFunctionOperatorFactory func(args []types.Operator, pool *pooling.LimitingPool) (types.InstantVectorOperator, error)
type instantVectorFunctionOperatorFactories func(args []types.Operator, pool *pooling.LimitingPool) (types.InstantVectorOperator, error)

func SimpleFunctionFactory(name string, seriesDataFunc functions.InstantVectorFunction) InstantVectorFunctionOperatorFactory {
// Simple Function Factory is for functions that have exactly 1 argument and drop the series name.
// transformationFunctionOperatorFactory creates an InstantVectorFunctionOperatorFactory for functions
// that have exactly 1 argument and drop the series name.
//
// Parameters:
// - name: The name of the function.
// - seriesDataFunc: The function to be wrapped.
//
// Returns:
//
// An InstantVectorFunctionOperatorFactory.
func transformationFunctionOperatorFactory(name string, seriesDataFunc functions.InstantVectorFunction) instantVectorFunctionOperatorFactories {
return func(args []types.Operator, pool *pooling.LimitingPool) (types.InstantVectorOperator, error) {
if len(args) != 1 {
// Should be caught by the PromQL parser, but we check here for safety.
Expand All @@ -37,7 +46,7 @@ func SimpleFunctionFactory(name string, seriesDataFunc functions.InstantVectorFu
}
}

func rateFunction(args []types.Operator, pool *pooling.LimitingPool) (types.InstantVectorOperator, error) {
func rateFunctionOperator(args []types.Operator, pool *pooling.LimitingPool) (types.InstantVectorOperator, error) {
if len(args) != 1 {
// Should be caught by the PromQL parser, but we check here for safety.
return nil, fmt.Errorf("expected exactly 1 argument for rate, got %v", len(args))
Expand All @@ -55,12 +64,12 @@ func rateFunction(args []types.Operator, pool *pooling.LimitingPool) (types.Inst
}, nil
}

var _ InstantVectorFunctionOperatorFactory = rateFunction
var _ instantVectorFunctionOperatorFactories = rateFunctionOperator

// These functions return an instant-vector.
var instantVectorFunctions = map[string]InstantVectorFunctionOperatorFactory{
"acos": SimpleFunctionFactory("acos", functions.Acos),
"histogram_count": SimpleFunctionFactory("histogram_count", functions.HistogramCount),
"histogram_sum": SimpleFunctionFactory("histogram_sum", functions.HistogramSum),
"rate": rateFunction,
var instantVectorFunctions = map[string]instantVectorFunctionOperatorFactories{
"acos": transformationFunctionOperatorFactory("acos", functions.Acos),
"histogram_count": transformationFunctionOperatorFactory("histogram_count", functions.HistogramCount),
"histogram_sum": transformationFunctionOperatorFactory("histogram_sum", functions.HistogramSum),
"rate": rateFunctionOperator,
}
5 changes: 3 additions & 2 deletions pkg/streamingpromql/functions/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ func DropSeriesName(seriesMetadata []types.SeriesMetadata, _ *pooling.LimitingPo

type InstantVectorFunction func(seriesData types.InstantVectorSeriesData, pool *pooling.LimitingPool) (types.InstantVectorSeriesData, error)

func FloatTransformationFunc(transform func(f float64) float64) InstantVectorFunction {
// floatTransformationFunc is not needed elsewhere, so it is not exported yet
func floatTransformationFunc(transform func(f float64) float64) InstantVectorFunction {
return func(seriesData types.InstantVectorSeriesData, pool *pooling.LimitingPool) (types.InstantVectorSeriesData, error) {
for i := range seriesData.Floats {
seriesData.Floats[i].F = transform(seriesData.Floats[i].F)
Expand All @@ -34,6 +35,6 @@ func FloatTransformationDropHistogramsFunc(transform func(f float64) float64) In
// https://prometheus.io/docs/prometheus/latest/querying/functions
pool.PutHPointSlice(seriesData.Histograms)
seriesData.Histograms = nil
return FloatTransformationFunc(transform)(seriesData, pool)
return floatTransformationFunc(transform)(seriesData, pool)
jhesketh marked this conversation as resolved.
Show resolved Hide resolved
}
}
2 changes: 1 addition & 1 deletion pkg/streamingpromql/functions/common_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ func TestDropSeriesName(t *testing.T) {

func TestFloatTransformationFunc(t *testing.T) {
transform := func(f float64) float64 { return f * 2 }
transformFunc := FloatTransformationFunc(transform)
transformFunc := floatTransformationFunc(transform)

seriesData := types.InstantVectorSeriesData{
Floats: []promql.FPoint{
Expand Down
22 changes: 10 additions & 12 deletions pkg/streamingpromql/testdata/ours/native_histograms.test
Original file line number Diff line number Diff line change
Expand Up @@ -30,26 +30,24 @@ load 1m
eval range from 0 to 5m step 1m mixed_metric
{__name__="mixed_metric"} 1 2 3 {{count:4 sum:5 buckets:[1 2 1]}} {{count:6 sum:8 buckets:[1 4 1]}} {{count:6 sum:8 buckets:[1 4 1]}}

eval range from 0 to 4m step 1m histogram_count(mixed_metric)
{} _ _ _ 4 6
eval instant at 3m histogram_count(mixed_metric)
{} 4

eval range from 0 to 4m step 1m histogram_sum(mixed_metric)
{} _ _ _ 5 8
eval instant at 4m histogram_sum(mixed_metric)
{} 8

# histogram_count ignores any float values
eval instant at 2m histogram_count(mixed_metric)
eval range from 0 to 5m step 1m histogram_count(mixed_metric)
{} _ _ _ 4 6 6

eval range from 0 to 5m step 1m histogram_sum(mixed_metric)
{} _ _ _ 5 8 8

# histogram_count ignores any float values
jhesketh marked this conversation as resolved.
Show resolved Hide resolved
eval instant at 3m histogram_count(mixed_metric)
{} 4
eval instant at 2m histogram_count(mixed_metric)

# histogram_sum ignores any float values
eval instant at 2m histogram_sum(mixed_metric)

# histogram_sum ignores any float values
eval instant at 4m histogram_sum(mixed_metric)
{} 8

clear

# Test multiple histograms
Expand Down