-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add size in bytes to batch processor #4761
Comments
This is a tricky problem. I've seen one library attempt to solve it in Go: https://godoc.org/google.golang.org/api/support/bundler It's an irritating problem because the natural solution is to repeatedly invoke The same issue affects the span batcher in every exporter across OpenTelemetry. Limiting a batch in terms of number of units leaves an unlimited batch size which immediately causes trouble when used with gRPC. |
Perhaps the size could be cached to some degree. The algorithm could work on the span sizes. The spans would be added to batch and once the size is higher as the threshold the flush would be called. The same could apply to other data types. |
This is a rather tricky problem and would love to work with others on coming up with a few solutions to evaluate. |
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
Is your feature request related to a problem? Please describe.
The Bulk processor controls the batch size via
send_batch_size
which is the number of spans or metrics. However the spans can vary in size and therefore it would be useful to instruct the batch processor to send the batch once a size in bytes is reached.Describe the solution you'd like
Add
send_batch_size_bytes
to the Batch processor https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/batchprocessor/README.md#batch-processorThe batch processor will send the batch once size in bytes is reached.
Describe alternatives you've considered
Additional context
In Jaeger we would like to put the batch processor in front of ES exporter that uses Elasticsearch bulk API (sends multiple requests in batch). Hence we would like to control how much data is sent to the storage.
Related to jaegertracing/jaeger#2295.
The text was updated successfully, but these errors were encountered: