Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/arm64: 63 Regressions on 4/17/2023 5:51:51 AM #16790

Closed
performanceautofiler bot opened this issue Apr 20, 2023 · 6 comments
Closed

[Perf] Windows/arm64: 63 Regressions on 4/17/2023 5:51:51 AM #16790

performanceautofiler bot opened this issue Apr 20, 2023 · 6 comments
Assignees
@EgorBo
Labels
arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 346.43 ns 386.62 ns 1.12 0.02 False
SerializeToWriter - Duration of single invocation 653.70 ns 691.89 ns 1.06 0.01 False
SerializeToStream - Duration of single invocation 537.44 ns 571.95 ns 1.06 0.01 False
SerializeToString - Duration of single invocation 485.24 ns 536.66 ns 1.11 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 445.17 ns 489.94 ns 1.10 0.03 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 386.6157758216629 > 362.04271550943366.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -20.05463924138319 (T) = (0 -382.4327315008801) / Math.Sqrt((57.78273616582408 / (16)) + (8.819904833784205 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12506166987832565 = (339.9215720701247 - 382.4327315008801) / 339.9215720701247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 691.8891556823024 > 685.5677924267005.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 3/17/2023 7:47:41 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -13.406919087528909 (T) = (0 -688.1250739213864) / Math.Sqrt((115.35263289427185 / (16)) + (6.946545535130817 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.05795029737042181 = (650.432327144053 - 688.1250739213864) / 650.432327144053 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToStream(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 571.9520398093322 > 551.6269995465412.
IsChangePoint: Marked as a change because one of 1/22/2023 4:26:49 PM, 3/18/2023 12:49:26 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -18.00984931938981 (T) = (0 -573.6745579136484) / Math.Sqrt((94.93793145288964 / (16)) + (22.459977082113724 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09864333263099168 = (522.166330850826 - 573.6745579136484) / 522.166330850826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 536.6591928940251 > 509.6100484425113.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -25.373059218344025 (T) = (0 -532.640582937591) / Math.Sqrt((45.71281864694944 / (16)) + (14.847303301785807 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11019779992164996 = (479.7708867511547 - 532.640582937591) / 479.7708867511547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 489.9404450942612 > 466.7282668150535.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 3/17/2023 1:30:52 PM, 4/12/2023 6:21:10 AM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -16.6476519133854 (T) = (0 -486.43213001310187) / Math.Sqrt((100.79487069005621 / (14)) + (10.514261305842028 / (12))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (12) - 2, .025) and -0.10773586995534239 = (439.12284796980714 - 486.43213001310187) / 439.12284796980714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 5.53 ms 6.09 ms 1.10 0.00 False
WriteDeepUtf8 - Duration of single invocation 5.20 ms 5.77 ms 1.11 0.00 False
WriteDeepUtf16 - Duration of single invocation 5.21 ms 5.86 ms 1.13 0.00 False
WriteDeepUtf8 - Duration of single invocation 5.56 ms 6.15 ms 1.11 0.00 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.089878392857143 > 5.798643443181818.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -47.809535847090814 (T) = (0 -6096298.9309385065) / Math.Sqrt((1439470219.7961 / (16)) + (584036956.834671 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.10561513322740142 = (5513943.096222642 - 6096298.9309385065) / 5513943.096222642 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.771695527728086 > 5.450405547872342.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -61.19284200930586 (T) = (0 -5823000.774995031) / Math.Sqrt((1100553012.148214 / (16)) + (397000565.63314986 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12290859927753284 = (5185640.913910078 - 5823000.774995031) / 5185640.913910078 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.864927777777778 > 5.482286630523733.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -47.124339652007855 (T) = (0 -5825689.05176173) / Math.Sqrt((1174212298.3386989 / (16)) + (1016067372.9097269 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11982917913560552 = (5202301.529826692 - 5825689.05176173) / 5202301.529826692 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.1498612121212135 > 5.831893083333334.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -29.185377805421613 (T) = (0 -6099091.379764545) / Math.Sqrt((2919449881.6955442 / (16)) + (2239272278.988687 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.10676492341373423 = (5510737.872819777 - 6099091.379764545) / 5510737.872819777 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAnyFiveValues - Duration of single invocation 9.60 ns 10.79 ns 1.12 0.09 False
IndexOfAnyFiveValues - Duration of single invocation 3.91 ns 5.09 ns 1.30 0.28 False
IndexOfAnyFourValues - Duration of single invocation 8.79 ns 10.64 ns 1.21 0.08 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfAnyFiveValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.794460984017551 > 10.05274372504155.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 2/9/2023 1:41:04 PM, 3/7/2023 9:28:02 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -31.25959581546483 (T) = (0 -10.91835797511539) / Math.Sqrt((0.017299472585865623 / (16)) + (0.0075006634064162005 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.13962594800780678 = (9.580650558371277 - 10.91835797511539) / 9.580650558371277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyFiveValues(Size: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.0874042802326604 > 4.317515659235891.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -24.866989452862356 (T) = (0 -5.09199273990236) / Math.Sqrt((0.01123515999395389 / (16)) + (0.009021385207956088 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.24317252539376724 = (4.095966276514599 - 5.09199273990236) / 4.095966276514599 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyFourValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.635303451926445 > 9.234152249587938.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -6.428742346818742 (T) = (0 -10.197416374386009) / Math.Sqrt((0.9082222189954232 / (16)) + (0.03323386136514836 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.18278120512787477 = (8.621557672861 - 10.197416374386009) / 8.621557672861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteStringsUtf8 - Duration of single invocation 69.58 ms 79.24 ms 1.14 0.00 True
WriteStringsUtf8 - Duration of single invocation 70.53 ms 79.25 ms 1.12 0.00 True
WriteStringsUtf8 - Duration of single invocation 69.47 ms 77.63 ms 1.12 0.01 True
WriteStringsUtf8 - Duration of single invocation 70.34 ms 79.73 ms 1.13 0.00 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: False, SkipValidation: False, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.23603571428572 > 73.07129076923077.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -55.812784893756486 (T) = (0 -78633298.16117218) / Math.Sqrt((54158261426.747314 / (16)) + (236953328798.75696 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.13224999233342685 = (69448707.17032968 - 78633298.16117218) / 69448707.17032968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: True, SkipValidation: True, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.25403833333333 > 73.7604935.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -53.332990278261654 (T) = (0 -79315519.92124543) / Math.Sqrt((60515570945.372955 / (16)) + (253819639268.50558 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12972896751661514 = (70207565.00171703 - 79315519.92124543) / 70207565.00171703 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: False, SkipValidation: True, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.63273653846153 > 72.76089.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -34.060965960419736 (T) = (0 -78273498.25457877) / Math.Sqrt((26968706662.319572 / (16)) + (690090057735.7645 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.13083857242266153 = (69217216.46520147 - 78273498.25457877) / 69217216.46520147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: True, SkipValidation: False, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.729795 > 73.8787532451923.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -57.497890750169766 (T) = (0 -79346580.89194138) / Math.Sqrt((37821104368.98685 / (16)) + (227069571064.04178 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12960919197095716 = (70242506.3959478 - 79346580.89194138) / 70242506.3959478 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 142.26 ns 165.08 ns 1.16 0.05 False
SerializeToString - Duration of single invocation 165.25 ns 183.33 ns 1.11 0.06 False
SerializeToUtf8Bytes - Duration of single invocation 144.73 ns 172.65 ns 1.19 0.06 False
SerializeToWriter - Duration of single invocation 59.95 ns 67.28 ns 1.12 0.08 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.07798818756322 > 149.7924973953407.
IsChangePoint: Marked as a change because one of 1/30/2023 9:42:46 PM, 3/13/2023 2:14:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -8.978170158466288 (T) = (0 -159.44060459928394) / Math.Sqrt((1.5387213283540242 / (16)) + (36.10713103796113 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12159930141907725 = (142.15469321134157 - 159.44060459928394) / 142.15469321134157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 183.33447631624037 > 174.9296212069031.
IsChangePoint: Marked as a change because one of 1/6/2023 8:18:17 PM, 1/9/2023 10:41:51 AM, 2/28/2023 5:58:27 PM, 3/6/2023 6:03:21 AM, 3/20/2023 7:45:43 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -11.771830167923259 (T) = (0 -181.57654854439585) / Math.Sqrt((4.793713083715359 / (16)) + (11.582145438549714 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.08492512402321102 = (167.3632073991045 - 181.57654854439585) / 167.3632073991045 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 172.6467468556998 > 151.53371720273634.
IsChangePoint: Marked as a change because one of 1/26/2023 2:33:00 AM, 3/13/2023 2:14:21 PM, 4/12/2023 6:21:10 AM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -4.286747541816714 (T) = (0 -153.33903915714046) / Math.Sqrt((0.643801474266888 / (14)) + (62.342196685668526 / (12))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (12) - 2, .025) and -0.0683776281153621 = (143.52513111645118 - 153.33903915714046) / 143.52513111645118 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 67.28324555959385 > 62.94186320955912.
IsChangePoint: Marked as a change because one of 2/3/2023 4:47:55 PM, 3/20/2023 7:45:43 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -10.388425959483724 (T) = (0 -65.26369790091127) / Math.Sqrt((0.34470617543994553 / (16)) + (2.5633463060229786 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09159396932832146 = (59.78752149122742 - 65.26369790091127) / 59.78752149122742 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 2.21 ms 2.75 ms 1.24 0.00 True
WriteBasicUtf16 - Duration of single invocation 2.22 ms 2.76 ms 1.24 0.00 True
WriteBasicUtf16 - Duration of single invocation 1.05 μs 1.12 μs 1.06 0.00 False
WriteBasicUtf16 - Duration of single invocation 3.02 ms 3.63 ms 1.20 0.00 True
WriteBasicUtf16 - Duration of single invocation 977.85 ns 1.07 μs 1.10 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.90 ms 3.49 ms 1.20 0.00 True
WriteBasicUtf16 - Duration of single invocation 2.06 ms 2.63 ms 1.28 0.00 True
WriteBasicUtf8 - Duration of single invocation 884.37 ns 978.03 ns 1.11 0.00 True
WriteBasicUtf8 - Duration of single invocation 2.89 ms 3.46 ms 1.20 0.00 True
WriteBasicUtf8 - Duration of single invocation 3.02 ms 3.63 ms 1.20 0.00 True
WriteBasicUtf8 - Duration of single invocation 2.07 ms 2.63 ms 1.27 0.00 True
WriteBasicUtf8 - Duration of single invocation 733.87 ns 811.04 ns 1.11 0.00 True
WriteBasicUtf8 - Duration of single invocation 682.01 ns 764.43 ns 1.12 0.00 True
WriteBasicUtf16 - Duration of single invocation 782.55 ns 862.30 ns 1.10 0.00 False
WriteBasicUtf8 - Duration of single invocation 942.43 ns 1.03 μs 1.09 0.00 True
WriteBasicUtf16 - Duration of single invocation 832.97 ns 922.19 ns 1.11 0.00 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.74966 > 2.3278817323008845.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -64.92295523442675 (T) = (0 -2747575.323565324) / Math.Sqrt((1062273628.0889825 / (16)) + (30544042.31334225 / (9))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (9) - 2, .025) and -0.24594030053766483 = (2205222.2906504055 - 2747575.323565324) / 2205222.2906504055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.755771512605042 > 2.325606537195797.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -66.54276992347577 (T) = (0 -2745556.607024869) / Math.Sqrt((984091197.6425436 / (16)) + (44972238.01411943 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.24517940607219213 = (2204948.615144129 - 2745556.607024869) / 2204948.615144129 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1170372588543598 > 1.0906530447758207.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -29.21229465697475 (T) = (0 -1116.6161358969657) / Math.Sqrt((68.65833328647753 / (16)) + (24.679843231875115 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.07297908868913874 = (1040.6690565248 - 1116.6161358969657) / 1040.6690565248 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.6305882125603866 > 3.167320120607787.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -61.144541970622775 (T) = (0 -3612166.0290951384) / Math.Sqrt((1200712988.4060564 / (16)) + (226604508.99059927 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.20094184873471163 = (3007777.631282351 - 3612166.0290951384) / 3007777.631282351 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.071511846914747 > 1030.2148851046015.
IsChangePoint: Marked as a change because one of 1/18/2023 3:44:12 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -38.5959638530333 (T) = (0 -1068.441489436203) / Math.Sqrt((30.480400990155218 / (16)) + (35.223418137354294 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09188916225347991 = (978.5255924979741 - 1068.441489436203) / 978.5255924979741 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.4853555555555564 > 3.0456964033235328.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -47.010849192158474 (T) = (0 -3458807.2647715313) / Math.Sqrt((1688246006.7589467 / (16)) + (420648201.74842316 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.19776934128951534 = (2887707.295169025 - 3458807.2647715313) / 2887707.295169025 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6297718796992484 > 2.164098598819324.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -74.13021758510158 (T) = (0 -2621802.600482487) / Math.Sqrt((834523974.1957514 / (16)) + (66721279.04090963 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.27692294649221566 = (2053219.1137176577 - 2621802.600482487) / 2053219.1137176577 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 978.0282747110577 > 927.6151883354962.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -38.36698083342631 (T) = (0 -978.0516878855129) / Math.Sqrt((60.60381918568194 / (16)) + (28.707970988339493 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11262476852689816 = (879.0489979658116 - 978.0516878855129) / 879.0489979658116 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.457149798792756 > 3.0278446068564553.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -55.82942826486444 (T) = (0 -3460555.7537862896) / Math.Sqrt((1156809277.361596 / (16)) + (372957565.4333791 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.2032165899086174 = (2876087.1341120005 - 3460555.7537862896) / 2876087.1341120005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.62931705882353 > 3.167608462962963.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -62.00537320233541 (T) = (0 -3614837.382852271) / Math.Sqrt((1141587906.0154066 / (16)) + (246559350.39451796 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.20202291827766308 = (3007294.892539858 - 3614837.382852271) / 3007294.892539858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6288119548872184 > 2.1634709147937623.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -77.57472885548445 (T) = (0 -2615816.501718238) / Math.Sqrt((790155073.8282857 / (16)) + (32996932.569604397 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.27429985957978015 = (2052748.0106455034 - 2615816.501718238) / 2052748.0106455034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 811.0416365409357 > 770.331174781587.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -46.6662866171236 (T) = (0 -810.1510216981719) / Math.Sqrt((45.66801976557107 / (16)) + (1.0182727817720392 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.1099228653964734 = (729.9165076744134 - 810.1510216981719) / 729.9165076744134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 764.4334204767833 > 716.7924052456851.
IsChangePoint: Marked as a change because one of 2/11/2023 12:14:55 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -32.657435123030425 (T) = (0 -758.4382831871823) / Math.Sqrt((88.23267307490485 / (16)) + (6.01265583118557 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11917548369886936 = (677.6759268176136 - 758.4382831871823) / 677.6759268176136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 862.29635745894 > 818.591082592886.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -52.279332916669944 (T) = (0 -861.984862237418) / Math.Sqrt((24.029836774786414 / (16)) + (10.48467252000541 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.10724375236886725 = (778.4960270882218 - 861.984862237418) / 778.4960270882218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0278147578039412 > 989.8924374788517.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -32.308951417826364 (T) = (0 -1024.2219705100676) / Math.Sqrt((96.69531340692451 / (16)) + (13.57899772146199 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09387551893659314 = (936.3240631856914 - 1024.2219705100676) / 936.3240631856914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 922.1905382631674 > 873.9972088996473.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -45.713955899561036 (T) = (0 -913.0664032285129) / Math.Sqrt((21.109463888104138 / (16)) + (19.51073779700096 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09955544719791604 = (830.3959618911007 - 913.0664032285129) / 830.3959618911007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfAnyValues - Duration of single invocation 17.40 ns 20.45 ns 1.18 0.09 False
IndexOfAnyFourValues - Duration of single invocation 31.02 ns 35.75 ns 1.15 0.06 False
IndexOfAnyFiveValues - Duration of single invocation 36.81 ns 42.08 ns 1.14 0.03 False
IndexOfAnyThreeValues - Duration of single invocation 22.89 ns 26.78 ns 1.17 0.11 False
IndexOfAnyThreeValues - Duration of single invocation 3.97 ns 5.15 ns 1.30 0.36 False
IndexOfAnyTwoValues - Duration of single invocation 17.45 ns 21.17 ns 1.21 0.09 False
IndexOfAnyFiveValues - Duration of single invocation 9.02 ns 12.02 ns 1.33 0.08 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.449786423511135 > 18.484876051951485.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -66.58800890035891 (T) = (0 -20.6182361380327) / Math.Sqrt((0.008201548048072885 / (16)) + (0.014363877359328371 / (9))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (9) - 2, .025) and -0.17412174666839428 = (17.560560645893464 - 20.6182361380327) / 17.560560645893464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.746395426804526 > 32.69046262290568.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -35.76433380925262 (T) = (0 -35.687812498811866) / Math.Sqrt((0.10975709273915041 / (16)) + (0.10055240343466001 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.14987039729275908 = (31.03637817169206 - 35.687812498811866) / 31.03637817169206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.08487769083617 > 38.76231947337339.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -20.817639041759943 (T) = (0 -41.40247423780578) / Math.Sqrt((0.158206631418575 / (16)) + (0.3985368593437601 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.12630517177944242 = (36.75955262852453 - 41.40247423780578) / 36.75955262852453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.78091470767028 > 23.909319624974994.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 2/1/2023 6:25:50 PM, 2/7/2023 3:56:56 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -19.628935152586163 (T) = (0 -26.670584942663293) / Math.Sqrt((0.5192031056531429 / (16)) + (0.019033969206641174 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.1579586112233732 = (23.032416430226338 - 26.670584942663293) / 23.032416430226338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyThreeValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.146136118619034 > 4.153252657486513.
IsChangePoint: Marked as a change because one of 3/17/2023 7:47:41 PM, 3/18/2023 3:40:27 PM, 4/12/2023 2:59:13 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -40.75252658681577 (T) = (0 -5.2670934990884986) / Math.Sqrt((0.005031665987993032 / (15)) + (0.008252505553185914 / (11))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (11) - 2, .025) and -0.3421683748009139 = (3.9243165000589255 - 5.2670934990884986) / 3.9243165000589255 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.168332188663385 > 18.325478244750947.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -14.50509567076386 (T) = (0 -21.159913070572077) / Math.Sqrt((0.8807720859394371 / (16)) + (0.0051315327938221056 / (9))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (9) - 2, .025) and -0.19284060143724885 = (17.739095269792614 - 21.159913070572077) / 17.739095269792614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Byte>.IndexOfAnyFiveValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.015109639209498 > 9.477736594430613.
IsChangePoint: Marked as a change because one of 3/16/2023 2:30:29 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -50.66017795178033 (T) = (0 -11.928716666018866) / Math.Sqrt((0.01707767092670371 / (16)) + (0.022643866274765515 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.324743439084623 = (9.004548589620812 - 11.928716666018866) / 9.004548589620812 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CheckArrayIsInterfaceNo - Duration of single invocation 35.97 μs 40.17 μs 1.12 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.17314965483235 > 37.76643522049071.
IsChangePoint: Marked as a change because one of 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -48.036302416505904 (T) = (0 -39926.55565924615) / Math.Sqrt((20771.82327972522 / (16)) + (55529.25553609952 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11059786854793374 = (35950.506290317906 - 39926.55565924615) / 35950.506290317906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 17.58 ns 20.24 ns 1.15 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.TryFormat(value: -2147483648)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.241259580881575 > 18.572840559684618.
IsChangePoint: Marked as a change because one of 1/21/2023 8:03:21 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -23.215458415760878 (T) = (0 -20.461696505285325) / Math.Sqrt((0.019752116296217558 / (16)) + (0.12486066737270976 / (9))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (9) - 2, .025) and -0.16205592633089125 = (17.608185666150916 - 20.461696505285325) / 17.608185666150916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 4.55 ms 5.10 ms 1.12 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>.DataContractJsonSerializer_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.100342777777778 > 4.788472440656566.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -5.53566530413187 (T) = (0 -5209617.191176469) / Math.Sqrt((73869919284.18034 / (16)) + (42503095040.50307 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.11118412690178127 = (4688347.381006958 - 5209617.191176469) / 4688347.381006958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterUInt64 - Duration of single invocation 2.25 ns 9.01 ns 4.00 0.77 True
FormatterInt32 - Duration of single invocation 1.94 ns 8.49 ns 4.38 0.61 False
FormatterDecimal - Duration of single invocation 46.24 ns 67.57 ns 1.46 0.00 True
FormatterUInt32 - Duration of single invocation 2.33 ns 8.25 ns 3.54 0.77 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.012605312346006 > 1.9670227021597253.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -49.44531652422075 (T) = (0 -9.011780840280563) / Math.Sqrt((0.2533354294704068 / (16)) + (0.04367078711721366 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -3.541801284202426 = (1.984186510234584 - 9.011780840280563) / 1.984186510234584 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.49397764212626 > 2.387234335669911.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -64.55333174606348 (T) = (0 -8.556266784671317) / Math.Sqrt((0.08925722252223998 / (16)) + (0.03783878958382363 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -2.7038504645892165 = (2.3101004931148776 - 8.556266784671317) / 2.3101004931148776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 67.57311052068947 > 48.58276719224687.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -106.73170925540383 (T) = (0 -67.26877114062283) / Math.Sqrt((0.19514191512808285 / (16)) + (0.2614499449684492 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.4507057268348155 = (46.369687453699825 - 67.26877114062283) / 46.369687453699825 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.24738070142657 > 2.417358660827902.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -45.23468351493213 (T) = (0 -8.527942605693688) / Math.Sqrt((0.21347747486071048 / (16)) + (0.0682700903968921 / (9))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (9) - 2, .025) and -3.298194326956805 = (1.9840756273417768 - 8.527942605693688) / 1.9840756273417768 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithCultureInfo - Duration of single invocation 120.63 ns 130.00 ns 1.08 0.00 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.ToStringWithCultureInfo(value: 12345, culture: zh)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.99665015212292 > 125.91547763185706.
IsChangePoint: Marked as a change because one of 2/2/2023 10:11:15 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -29.95388521352437 (T) = (0 -131.6629327569699) / Math.Sqrt((0.5990112224898472 / (16)) + (1.180417402509296 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09854751240158278 = (119.8518327797546 - 131.6629327569699) / 119.8518327797546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Base64DecodeDestinationTooSmall - Duration of single invocation 314.92 ns 336.63 ns 1.07 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Base64Tests.Base64DecodeDestinationTooSmall(NumberOfBytes: 1000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.634155476139 > 331.6030621612162.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 2/1/2023 6:25:50 PM, 2/7/2023 3:56:56 PM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -101.40398509470508 (T) = (0 -336.28899394385496) / Math.Sqrt((0.2936608447637834 / (16)) + (0.22387912211410776 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.06480874813286563 = (315.8210284556126 - 336.28899394385496) / 315.8210284556126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_OneArg - Duration of single invocation 1.07 μs 1.14 μs 1.06 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_OneArg(s: "Testing {0}, {0:C}, {0:E} - {0:F4}{0:G}{0:N} , !!", o: 3.14159)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.137037604370041 > 1.126502690648954.
IsChangePoint: Marked as a change because one of 3/2/2023 11:15:04 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -20.317566286107226 (T) = (0 -1133.5162041825838) / Math.Sqrt((105.74448638959896 / (16)) + (27.210785428337967 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.057921634428740626 = (1071.4557366951503 - 1133.5162041825838) / 1071.4557366951503 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithCultureInfo - Duration of single invocation 123.70 ns 134.38 ns 1.09 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Single.ToStringWithCultureInfo(value: 12345, culture: zh)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.38027124249712 > 128.90290405466544.
IsChangePoint: Marked as a change because one of 3/17/2023 5:40:49 AM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -26.172396719072605 (T) = (0 -134.55068971129032) / Math.Sqrt((0.6597951748584818 / (16)) + (1.669956876223484 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09740940815734614 = (122.60755986884932 - 134.55068971129032) / 122.60755986884932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 112.81 ns 124.36 ns 1.10 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.36236076126829 > 118.51221280244017.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 3/20/2023 7:45:43 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -9.888568537688808 (T) = (0 -120.49955819343984) / Math.Sqrt((0.7821715543434301 / (16)) + (5.675004934290403 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.06886478591105241 = (112.73601654930695 - 120.49955819343984) / 112.73601654930695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.Perf_NumberCultureInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 138.58 ns 147.35 ns 1.06 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.Perf_NumberCultureInfo*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: )

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.35460230385377 > 143.84776069373683.
IsChangePoint: Marked as a change because one of 2/17/2023 12:28:21 PM, 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -16.41806962265399 (T) = (0 -147.1024046656597) / Math.Sqrt((0.8634767356543814 / (16)) + (3.5350878079085146 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.07671002714873441 = (136.62211826447427 - 147.1024046656597) / 136.62211826447427 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 73.52 ns 78.23 ns 1.06 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Decimal.ToString(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 78.22594287763896 > 76.72756755212781.
IsChangePoint: Marked as a change because one of 1/31/2023 1:54:46 PM, 2/17/2023 12:28:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -16.411890770526536 (T) = (0 -77.85874133313688) / Math.Sqrt((0.35595023857538505 / (16)) + (0.5162886907845973 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.060775078005438535 = (73.39797375286535 - 77.85874133313688) / 73.39797375286535 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 32.68 ns 35.40 ns 1.08 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.395580255618455 > 34.63458758023484.
IsChangePoint: Marked as a change because one of 1/20/2023 9:27:13 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -18.791853862870013 (T) = (0 -35.4402433466895) / Math.Sqrt((0.06668012272122835 / (16)) + (0.13778229767342728 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.07646325763118184 = (32.92285463107933 - 35.4402433466895) / 32.92285463107933 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_DifferentFirstChar - Duration of single invocation 13.39 ns 14.58 ns 1.09 0.11 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.580858994596209 > 14.061998995811484.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -53.970203590005006 (T) = (0 -14.666369466862562) / Math.Sqrt((0.00392675666121162 / (16)) + (0.0035551475859809728 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.09915255233750697 = (13.343342956055013 - 14.666369466862562) / 13.343342956055013 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.ReadJson<TreeRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 45.79 μs 51.34 μs 1.12 0.34 False
DeserializeFromString - Duration of single invocation 37.56 μs 43.03 μs 1.15 0.38 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;TreeRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<TreeRecord>.DeserializeFromReader(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.338398344313326 > 48.67958806123464.
IsChangePoint: Marked as a change because one of 2/18/2023 8:36:57 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -5.245239734725174 (T) = (0 -50237.94585388069) / Math.Sqrt((1596463.9695270904 / (16)) + (3255444.9059710545 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.07306676812060647 = (46817.166784382454 - 50237.94585388069) / 46817.166784382454 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.ReadJson<TreeRecord>.DeserializeFromString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.032529843893485 > 38.847405675286666.
IsChangePoint: Marked as a change because one of 3/4/2023 1:07:20 AM, 4/17/2023 2:42:45 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -7.224012426661512 (T) = (0 -42338.03460786136) / Math.Sqrt((3700322.152242307 / (18)) + (1752709.651856817 / (8))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (8) - 2, .025) and -0.12510109791464513 = (37630.42688904504 - 42338.03460786136) / 37630.42688904504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAnyFourValues - Duration of single invocation 6.90 ns 8.72 ns 1.26 0.16 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.72382291146058 > 7.244530457968424.
IsChangePoint: Marked as a change because one of 2/9/2023 1:41:04 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -21.711470311245648 (T) = (0 -8.75769032477167) / Math.Sqrt((0.08676418138467902 / (16)) + (0.01654217971369719 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.2635139836073949 = (6.931217571306991 - 8.75769032477167) / 6.931217571306991 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EncodeUtf8 - Duration of single invocation 70.24 ns 76.23 ns 1.09 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

Histogram

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,no <escaping /> required,512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 76.22694779173804 > 74.92606543536446.
IsChangePoint: Marked as a change because one of 1/11/2023 1:25:06 PM, 1/21/2023 9:40:41 PM, 2/1/2023 6:25:50 PM, 2/7/2023 3:56:56 PM, 2/14/2023 11:21:41 AM, 3/10/2023 4:14:40 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 5:25:38 AM falls between 4/11/2023 2:19:19 PM and 4/20/2023 5:25:38 AM.
IsRegressionStdDev: Marked as regression because -18.812087767314587 (T) = (0 -75.64557806539622) / Math.Sqrt((0.30585117276853124 / (16)) + (0.3987280311762316 / (10))) is less than -2.0638985616205927 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (10) - 2, .025) and -0.06428270391984213 = (71.07658311723684 - 75.64557806539622) / 71.07658311723684 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kunalspathak
Copy link
Member

Could be from dotnet/runtime#84587 but @cincuranet to double check from the data explorer?

@cincuranet
Copy link

cincuranet commented Apr 26, 2023 •
edited
Loading

@kunalspathak

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: SourceGen)

The regression first occurred between dotnet/runtime@e4743ec...ed09ae5. Although looks like it's back to similar numbers: ~347 ns before, ~380 ns when in regression and now ~350 ns from build 20230425.3.

System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)

The regression first occurred between dotnet/runtime@ed0f1c5...ed09ae5.

Some others you want me to check?

@kunalspathak
Copy link
Member

possibly dotnet/runtime#85277 fixed some of them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EgorBo EgorBo

Labels
arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@EgorBo @cincuranet @kunalspathak