Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Tail Based Sampling Processor From OTEL Collector Extension #5878

Merged
merged 30 commits into from
Aug 31, 2024
Merged

Support Tail Based Sampling Processor From OTEL Collector Extension #5878

merged 30 commits into from
Aug 31, 2024

Conversation

mahadzaryab1
Copy link
Contributor

@mahadzaryab1 mahadzaryab1 commented Aug 22, 2024 •
edited
Loading

Which problem is this PR solving?

Description of the changes

  • Added the tail-based sampling processor extension from otel to jaeger
  • Added a docker compose to demonstrate usage of the tail-based sampling processor extension in jaeger.
  • Added an end to end integration test to test that the new processor works as expected
  • Added a README to the docker compose setup describing the setup and usage of the new processor

How was this change tested?

  • An end to end integration test was added and is run from the CI

Checklist

@mahadzaryab1 mahadzaryab1 mentioned this pull request Aug 22, 2024
Closed
4 tasks
@codecov Codecov
Copy link

codecov bot commented Aug 23, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.82%. Comparing base (9a30dfc) to head (c50ded3).
Report is 1 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #5878      +/-   ##
==========================================
- Coverage   96.83%   96.82%   -0.02%     
==========================================
  Files         342      342              
  Lines       16524    16525       +1     
==========================================
- Hits        16001    16000       -1     
- Misses        337      339       +2     
  Partials      186      186
Flag Coverage Δ
badger_v1 8.05% <ø> (ø)
badger_v2 1.82% <ø> (ø)
cassandra-3.x-v1 16.62% <ø> (ø)
cassandra-3.x-v2 1.75% <ø> (ø)
cassandra-4.x-v1 16.62% <ø> (ø)
cassandra-4.x-v2 1.75% <ø> (ø)
elasticsearch-6.x-v1 18.79% <ø> (+0.01%) ⬆️
elasticsearch-7.x-v1 18.85% <ø> (+0.01%) ⬆️
elasticsearch-8.x-v1 19.03% <ø> (ø)
elasticsearch-8.x-v2 1.82% <ø> (ø)
grpc_v1 9.49% <ø> (ø)
grpc_v2 7.16% <ø> (ø)
kafka-v1 9.74% <ø> (ø)
kafka-v2 1.82% <ø> (ø)
memory_v2 1.82% <ø> (ø)
opensearch-1.x-v1 18.90% <ø> (+0.01%) ⬆️
opensearch-2.x-v1 18.90% <ø> (ø)
opensearch-2.x-v2 1.82% <ø> (ø)
tailsampling-processor 0.46% <ø> (?)
unittests 95.30% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yurishkuro
Copy link
Member

I tested the binary sizes

  • without this change - 75Mb
  • with this change - 114Mb
  • with only tail sampler (without load balancer) - 78Mb

I would suggest we only include tail sampler. That's the component that's most useful in the final collector, which needs to be Jaeger. But the upstream load balancing collectors can be just OTEL Collectors.

@mahadzaryab1 mahadzaryab1 force-pushed the tail-based-sampling branch 3 times, most recently from 84a9b93 to 929c409 Compare August 24, 2024 23:28
@mahadzaryab1 mahadzaryab1 force-pushed the tail-based-sampling branch 2 times, most recently from 01c9743 to e15af38 Compare August 30, 2024 23:47
yurishkuro
yurishkuro reviewed Aug 31, 2024
View reviewed changes
cmd/jaeger/internal/integration/tailsampling_test.go Outdated Show resolved Hide resolved
cmd/jaeger/internal/integration/tailsampling_test.go Outdated Show resolved Hide resolved
docker-compose/tail-sampling/docker-compose.yml Show resolved Hide resolved
docker-compose/tail-sampling/Makefile Outdated Show resolved Hide resolved
docker-compose/tail-sampling/Makefile Outdated Show resolved Hide resolved
cmd/jaeger/internal/integration/tailsampling_test.go Outdated Show resolved Hide resolved
Makefile
@@ -170,6 +170,10 @@ index-cleaner-integration-test: docker-images-elastic
index-rollover-integration-test: docker-images-elastic
$(MAKE) storage-integration-test COVEROUT=cover-index-rollover.out

.PHONY: tail-sampling-integration-test
tail-sampling-integration-test:
SAMPLING=tail $(MAKE) jaeger-v2-storage-integration-test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this runs go test, but when do you start the docker compose environment?

All other e2e tests have a driver script that orchestrates all components of the test, e.g. scripts/es-integration-test.sh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not using the docker-compose environment for my test. Calling e2eInitialize is enough to start the Jaeger collector. You can simply run this test by calling make tail-sampling-integration-test. Let me know if you want to change any of this setup though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. However, it means that the new docker compose file will begin to rot since it's not being exercised by the CI, something we tried to avoid (e.g. see e2e spm test). So it would be good to actually combine using docker compose with e2e test.

Copy link
Contributor Author

@mahadzaryab1 mahadzaryab1 Aug 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see! What would you want this to look like? The current docker-compose set up generates load using tracegen which ideally we wouldn't want in the integration test so we can manually generate those. And the existing setup in the E2E tests does some nice things for us like flush the storage in between tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that's why I was asking from the start what your plan would be. The way you are using e2e_integration framework is very lightweight, and I could easily see an alternative setup where everything is just orchestrated from a shell script

  • run docker-compose with one config
    • maybe don't include tracegen in compose, run it manually
  • do a curl against query service to retrieve service names as JSON (trivial to write)
  • shut down docker-compose (to clear the storage) and run again with different config

If you are interested to pursue this approach, I would suggest still merging this PR first so that we already have something in place. Can you finish the README?

Copy link
Contributor Author

@mahadzaryab1 mahadzaryab1 Aug 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro That sounds good to me and I can pursue that approach in a follow-up PR. And yes, working on the README now. Will push it up soon.

cmd/jaeger/config-tail-sampling-service-name-policy.yaml Show resolved Hide resolved
@yurishkuro
Copy link
Member

yurishkuro commented Aug 31, 2024 •
edited
Loading

make sure to do git pull, I pushed updates to go.mod to resolve conflicts

@mahadzaryab1
Copy link
Contributor Author

make sure to do git pull, I pushed updates to go.mod to resolve conflicts

Thank you so much for doing this for me!

@yurishkuro yurishkuro added changelog:exprimental Change to an experimental part of the code v2 labels Aug 31, 2024
@mahadzaryab1
Add Dependencies
e55ddca 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Include Load Balancing Exporter And Tail Sampling Processor
822da09 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Remove Load Balancing Exporter
1d08303 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Set Up Tail Sampling Example
29d64f0 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Add Makefile To Example
191252c 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Use Tracegen And Simplify Config
2cde4aa 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Add Makefile Target For Running Tail Sampling Integration Test
f5d5d54 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Add Jaeger Config Files For Tail Sampling Integration Test
776e054 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Flip Policies In Config Files
15a5eaf 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Add E2E Integration Test For Tail Sampling Processor
c7e29d1 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
yurishkuro and others added 2 commits August 31, 2024 01:39
@yurishkuro
update go.mod
124ca49 
Signed-off-by: Yuri Shkuro <github@ysh.us>
@yurishkuro
Merge branch 'main' into tail-based-sampling
bd6fd79
@mahadzaryab1
Make Test Logging Unconditional
91ae10c 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Fix Docker Compose Setup
dc9ab9b 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Remove Endpoints From Config That Are Not Needed
1fb1d65 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Get Combined Output For Tracegen
6a00d65 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
More Cleanup For Docker Setup
49a9c03 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1
Address Feedback From PR Review
603fb63 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1 mahadzaryab1 changed the title [WIP] feat: support tail based sampling from otel collector [WIP] feat: support tail based sampling from otel collector extension Aug 31, 2024
@mahadzaryab1
Update README Documenting Setup Of Tail Sampling Processor
c50ded3 
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
@mahadzaryab1 mahadzaryab1 changed the title [WIP] feat: support tail based sampling from otel collector extension feat(processor): support tail based sampling processor from otel collector extension Aug 31, 2024
@mahadzaryab1 mahadzaryab1 changed the title feat(processor): support tail based sampling processor from otel collector extension Support Tail Based Sampling Processor From Otel Collector Extension Aug 31, 2024
@mahadzaryab1 mahadzaryab1 marked this pull request as ready for review August 31, 2024 16:47
@mahadzaryab1 mahadzaryab1 requested a review from a team as a code owner August 31, 2024 16:47
@dosubot dosubot bot added area/otel docker Pull requests that update Docker code enhancement labels Aug 31, 2024
@mahadzaryab1
Copy link
Contributor Author

mahadzaryab1 commented Aug 31, 2024 •
edited
Loading

@yurishkuro the README and the rest of the PR is ready for review now

yurishkuro
yurishkuro approved these changes Aug 31, 2024
View reviewed changes
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@mahadzaryab1
Copy link
Contributor Author

@yurishkuro - there looks to be a failing test in the CI. Is it a flaky test? I don't believe its related to my changes.

@mahadzaryab1 mahadzaryab1 changed the title Support Tail Based Sampling Processor From Otel Collector Extension Support Tail Based Sampling Processor From OTEL Collector Extension Aug 31, 2024
@yurishkuro yurishkuro merged commit 8ad6ed0 into jaegertracing:main Aug 31, 2024
50 checks passed
@yurishkuro
Copy link
Member

🎉 🎉 🎉

@mahadzaryab1 mahadzaryab1 deleted the tail-based-sampling branch August 31, 2024 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yurishkuro yurishkuro yurishkuro approved these changes

Assignees
No one assigned
Labels
area/otel changelog:exprimental Change to an experimental part of the code docker Pull requests that update Docker code enhancement v2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support tail-based sampling from OTEL Collector
2 participants
@mahadzaryab1 @yurishkuro