Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

plugin/storage/badger/spanstore: BenchmarkServiceIndexLimitFetch crashes #3301

Closed
odeke-em opened this issue Oct 4, 2021 · 2 comments · Fixed by #3325
Closed

plugin/storage/badger/spanstore: BenchmarkServiceIndexLimitFetch crashes #3301

odeke-em opened this issue Oct 4, 2021 · 2 comments · Fixed by #3325
Labels
bug

Comments

@odeke-em
Copy link
Contributor

odeke-em commented Oct 4, 2021

Describe the bug
A clear and concise description of what the bug is.
I tried to continuously benchmark Jaeger using Orijtech Inc's continuous benchmarking product "Bencher" per https://dashboard.github.orijtech.com/?repo=orijtech/jaeger but it failed each time
Screen Shot 2021-10-04 at 1 32 34 AM

To Reproduce
Steps to reproduce the behavior:

cd plugin/storage/badger/spanstore && go test -run=^$ -bench=.
badger 2021/10/04 01:29:59 INFO: All 0 tables opened in 0s
badger 2021/10/04 01:29:59 INFO: Discard stats nextEmptySlot: 0
badger 2021/10/04 01:29:59 INFO: Set nextTxnTs to 0
badger 2021/10/04 01:30:05 INFO: Lifetime L0 stalled for: 0s
badger 2021/10/04 01:30:05 INFO: 
Level 0 [ ]: NumTables: 04. Size: 34 MiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level Done
goos: darwin
goarch: amd64
pkg: github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore
cpu: Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
BenchmarkWrites-8                                 	       1	5548476737 ns/op
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3f0 pc=0x1447a8f]

goroutine 32060 [running]:
github.com/dgraph-io/badger/v3.(*DB).IsClosed(...)
	/Users/emmanuelodeke/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:537
github.com/dgraph-io/badger/v3.(*DB).Update(0xc004a44583, 0xc00ac5f6c8)
	/Users/emmanuelodeke/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/txn.go:813 +0x2f
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore.(*SpanWriter).WriteSpan(0xc004a445a0, {0x159750b1e, 0x1c8a800}, 0xc0000dc0f0)
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/writer.go:105 +0xb6f
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore_test.writeSpans({0x17f1c60, 0xc004a445a0}, {0xc0000a2000, 0x40, 0x40}, {0xc0000ae200, 0x20, 0xc00ac5fba8}, {0xc0000ae000, 0x20, ...}, ...)
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/read_write_test.go:492 +0xc2
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore_test.makeReadBenchmark.func1({0xc00ac5fd30, 0x0}, {0x17f1c60, 0xc004a445a0}, {0x1807298, 0xc007292810})
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/read_write_test.go:559 +0x1a5
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore_test.runLargeFactoryTest({0x1813d70, 0xc004a2eb40}, 0xc00ac5fe60)
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/read_write_test.go:646 +0x757
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore_test.makeReadBenchmark(0x1668600, {0xc000186ed0, 0x17169ec, 0x2}, 0x100000001, {0x1728098, 0x106553c})
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/read_write_test.go:543 +0x5f
github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore_test.BenchmarkServiceTagsRangeQueryLimitIndexFetch(0xc004a2eb40)
	/Users/emmanuelodeke/go/src/github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore/read_write_test.go:594 +0x18f
testing.(*B).runN(0xc004a2eb40, 0x1)
	/Users/emmanuelodeke/go/src/go.googlesource.com/go/src/testing/benchmark.go:192 +0x102
testing.(*B).run1.func1()
	/Users/emmanuelodeke/go/src/go.googlesource.com/go/src/testing/benchmark.go:232 +0x59
created by testing.(*B).run1
	/Users/emmanuelodeke/go/src/go.googlesource.com/go/src/testing/benchmark.go:225 +0x9c
exit status 2
FAIL	github.com/jaegertracing/jaeger/plugin/storage/badger/spanstore	6.080s

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

  • OS: [e.g. Linux]
  • Jaeger version: [e.g. 1.8]
  • Deployment: [e.g. bare metal, Docker, Kubernetes, ...]

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.

Additional context
Add any other context about the problem here.

I'd highly recommend adding continuous benchmarking to Jaeger so that things stay performant on every PR/commit without y'all even sweating at all, we handle all the heavy lifting for you. /cc-ing my colleagues @kirbyquerby @cuonglm.
@odeke-em odeke-em added the bug label Oct 4, 2021
@jpkrohling
Copy link
Contributor

Our badger implementation could really make use of some love. Are you interested in contributing a PR to get it better? Continuous benchmarking also sounds interesting :-)

@odeke-em odeke-em mentioned this issue Oct 11, 2021
Closed
@odeke-em
Copy link
Contributor Author

@jpkrohling sure I've mailed this ifx #3320

pavolloffay pushed a commit to pavolloffay/jaeger that referenced this issue Oct 14, 2021
@odeke-em @pavolloffay
plugin/storage/badger/spanstore: use (TB).TempDir not hardcoded non-p…
26bc987 
…ortable /mnt/*

Previously the Benchmark code ALWAYS assumed that "/mnt/ssd/" would
exist, yet when there is no permission or on read only systems,
using it would fail. This change instead uses the benchmarks'
temporary directory as the base for operations.

Fixes jaegertracing#3301

Signed-off-by: Emmanuel T Odeke <emmanuel@orijtech.com>
@pavolloffay pavolloffay mentioned this issue Oct 14, 2021
Merged
yurishkuro pushed a commit that referenced this issue Oct 14, 2021
@pavolloffay @odeke-em
[badger]: use (TB).TempDir instead of non-portable /mnt/* (#3325)
51ce980 
Previously the Benchmark code ALWAYS assumed that "/mnt/ssd/" would
exist, yet when there is no permission or on read only systems,
using it would fail. This change instead uses the benchmarks'
temporary directory as the base for operations.

Fixes #3301

Signed-off-by: Emmanuel T Odeke <emmanuel@orijtech.com>

Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
2 participants
@jpkrohling @odeke-em