Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Statistically tests should be able to compare methods only among a specific runtime in multi-framework bench #2585

Open
tats-u opened this issue Jul 2, 2024 · 0 comments
Assignees

Comments

@tats-u
Copy link

tats-u commented Jul 2, 2024

Method Runtime Input Count Mean Error StdDev Ratio MannWhitney(5%) RatioSD Gen0 Gen1 Gen2 Allocated Alloc Ratio
RepeatWithCounter .NET 8.0 👍 3 22.51 ns 0.477 ns 0.423 ns 1.55 Slower 0.03 0.0204 - - 128 B 3.20
RepeatNoCounter .NET 8.0 👍 3 14.51 ns 0.219 ns 0.194 ns 1.00 Base 0.00 0.0063 - - 40 B 1.00
RepeatPowerShell .NET 8.0 👍 3 16.76 ns 0.158 ns 0.132 ns 1.16 Slower 0.02 0.0063 - - 40 B 1.00
RepeatDoubleBlockSize .NET 8.0 👍 3 17.74 ns 0.304 ns 0.455 ns 1.22 Slower 0.04 0.0063 - - 40 B 1.00
RepeatWithCounter NativeAOT 8.0 👍 3 27.16 ns 0.508 ns 0.475 ns 1.87 Slower 0.02 0.0179 - - 112 B 2.80
RepeatNoCounter NativeAOT 8.0 👍 3 17.75 ns 0.074 ns 0.058 ns 1.22 Slower 0.02 0.0063 - - 40 B 1.00
RepeatPowerShell NativeAOT 8.0 👍 3 19.38 ns 0.435 ns 0.565 ns 1.34 Slower 0.04 0.0063 - - 40 B 1.00
RepeatDoubleBlockSize NativeAOT 8.0 👍 3 20.51 ns 0.142 ns 0.133 ns 1.41 Slower 0.02 0.0063 - - 40 B 1.00

For example, we should be able to compare 4 methods in each runtime respectly:

Method Runtime MannWhitney(5%)
RepeatWithCounter NET 8.0 Slower
RepeatNoCounter .NET 8.0 Base
RepeatPowerShell .NET 8.0 Slower
RepeatDoubleBlockSize .NET 8.0 Slower
RepeatWithCounter NativeAOT 8.0 ? (Slower?)
RepeatNoCounter NativeAOT 8.0 Base
RepeatPowerShell NativeAOT 8.0 ? (Slower?)
RepeatDoubleBlockSize NativeAOT 8.0 ? (Slower?)
@AndreyAkinshin AndreyAkinshin self-assigned this Jul 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants