Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 26 Regressions on 5/2/2023 9:04:55 PM #85989

Closed
performanceautofiler bot opened this issue May 9, 2023 · 17 comments
Closed

[Perf] Linux/x64: 26 Regressions on 5/2/2023 9:04:55 PM #85989

performanceautofiler bot opened this issue May 9, 2023 · 17 comments
Assignees
@AndyAyersMS
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 23.76 ns 34.01 ns 1.43 0.03 False
ParseSpan - Duration of single invocation 23.26 ns 34.00 ns 1.46 0.03 False
Parse - Duration of single invocation 11.86 ns 15.23 ns 1.28 0.09 False
TryParse - Duration of single invocation 12.63 ns 17.11 ns 1.35 0.05 False
Parse - Duration of single invocation 23.17 ns 35.23 ns 1.52 0.03 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.01435809036666 > 25.3743801477778.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -47.04559135093131 (T) = (0 -34.297548225823824) / Math.Sqrt((0.8005490307930864 / (23)) + (0.31126691835768755 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.44004695426090074 = (23.816965220711797 - 34.297548225823824) / 23.816965220711797 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.0025681010384 > 24.611768631477045.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -46.84627991199859 (T) = (0 -34.268259827484265) / Math.Sqrt((0.8428960124932353 / (23)) + (0.27599784156844087 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.43893043423743877 = (23.815091412424486 - 34.268259827484265) / 23.815091412424486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.233663593779918 > 12.463789074068497.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -17.55416495146352 (T) = (0 -15.950257694152228) / Math.Sqrt((0.56611465688463 / (23)) + (0.459656587047386 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.3111793848142055 = (12.164817323155507 - 15.950257694152228) / 12.164817323155507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.TryParse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.110386249177225 > 13.017706441037802.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -21.476102037711637 (T) = (0 -16.274352469310102) / Math.Sqrt((0.20947989456445193 / (23)) + (0.46853402978353403 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.3053286466629373 = (12.467628371533376 - 16.274352469310102) / 12.467628371533376 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.228765884344334 > 24.448686000325765.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -30.66097569867656 (T) = (0 -34.29792558643127) / Math.Sqrt((1.8703417116844718 / (23)) + (0.6842109628104064 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.4320577592674471 = (23.9500993339654 - 34.29792558643127) / 23.9500993339654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 538.46 ns 686.08 ns 1.27 0.00 True
Compare_Same_Upper - Duration of single invocation 1.15 μs 2.28 μs 1.98 0.36 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 686.0821531424299 > 564.3787270447281.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -579.3427571726128 (T) = (0 -685.6942626411378) / Math.Sqrt((0.725414759068659 / (23)) + (0.714626546587564 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.2760794439870314 = (537.3444936145421 - 685.6942626411378) / 537.3444936145421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.27794682493587 > 1.1776218656975188.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -93.48272309886674 (T) = (0 -2250.1618006414624) / Math.Sqrt((2809.1135448678656 / (23)) + (636.5590553633673 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -1.0531923371231457 = (1095.9332742271495 - 2250.1618006414624) / 1095.9332742271495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 1.57 μs 1.67 μs 1.06 0.38 False
HashSet - Duration of single invocation 8.88 μs 9.66 μs 1.09 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.Stack(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6693939692714146 > 1.6459481093363364.
IsChangePoint: Marked as a change because one of 3/2/2023 4:26:55 AM, 4/9/2023 1:58:52 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -16.106220302535547 (T) = (0 -1729.8309426630458) / Math.Sqrt((50.09530921594631 / (23)) + (2164.523623082486 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.10560978515321734 = (1564.5944580920318 - 1729.8309426630458) / 1564.5944580920318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.AddGivenSize<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.659812924644845 > 9.415499566867954.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -14.906857164051546 (T) = (0 -9967.163168512167) / Math.Sqrt((5329.6980621772145 / (23)) + (89724.65834760114 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.11157541090470617 = (8966.699938423375 - 9967.163168512167) / 8966.699938423375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 13.44 ns 16.83 ns 1.25 0.08 False
Parse - Duration of single invocation 13.24 ns 16.24 ns 1.23 0.12 False
Parse - Duration of single invocation 11.03 ns 12.15 ns 1.10 0.29 False
TryParse - Duration of single invocation 14.55 ns 15.74 ns 1.08 0.08 False
TryParse - Duration of single invocation 10.57 ns 12.09 ns 1.14 0.32 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.Parse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.826146143964596 > 14.119530080196824.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 3/13/2023 6:11:08 PM, 4/17/2023 1:54:35 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -46.76185571127924 (T) = (0 -16.518261848600194) / Math.Sqrt((0.02085848296423869 / (23)) + (0.07074501304336984 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.2271602380239204 = (13.460558235817132 - 16.518261848600194) / 13.460558235817132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.23877566364392 > 13.90532514864634.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 3/13/2023 12:04:12 PM, 4/17/2023 6:40:18 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -48.36895243368514 (T) = (0 -16.200777365552934) / Math.Sqrt((0.005068269745208753 / (23)) + (0.07270830373925728 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.22127048370204414 = (13.265511270233459 - 16.200777365552934) / 13.265511270233459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.Parse(value: "0")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.15398815823343 > 10.635719684883263.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -18.676969945448686 (T) = (0 -12.135074364221301) / Math.Sqrt((0.14678046554799734 / (23)) + (0.06144337488222862 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.17437792912437924 = (10.333193483352723 - 12.135074364221301) / 10.333193483352723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.TryParse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.739083277852119 > 15.320294620178725.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -16.26890816159469 (T) = (0 -15.79932298951513) / Math.Sqrt((0.034046559937457296 / (23)) + (0.0682911959451056 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.07623619696671498 = (14.680163177975475 - 15.79932298951513) / 14.680163177975475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.TryParse(value: "0")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.085731777599758 > 10.746377702684738.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -8.268523248929515 (T) = (0 -11.909829426312127) / Math.Sqrt((0.5631650447733183 / (23)) + (0.09868101589272804 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.13456361094530517 = (10.49727781802291 - 11.909829426312127) / 10.49727781802291 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt32 - Duration of single invocation 5.68 ns 10.58 ns 1.86 0.04 False
FormatterInt32 - Duration of single invocation 1.55 ns 7.09 ns 4.56 0.17 False
FormatterInt32 - Duration of single invocation 4.45 ns 8.83 ns 1.99 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.575831831229376 > 5.957003043233291.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1574.8660868034437 (T) = (0 -10.562699249205188) / Math.Sqrt((0.00012342313623343488 / (23)) + (8.982280015006975E-05 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.8622160891109447 = (5.672112549649386 - 10.562699249205188) / 5.672112549649386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.08956926702956 > 1.6324580612478683.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -3087.9906465206955 (T) = (0 -7.088097218182055) / Math.Sqrt((4.623066496265888E-06 / (23)) + (6.317721837668212E-05 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -3.5553670501149837 = (1.555988165213414 - 7.088097218182055) / 1.555988165213414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.830698653458 > 4.6845527687024315.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -118.10554486920714 (T) = (0 -8.822609142330025) / Math.Sqrt((0.03070856775007168 / (23)) + (0.00021011967589717185 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.9645442995820404 = (4.490918908882354 - 8.822609142330025) / 4.490918908882354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 27.24 μs 30.06 μs 1.10 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.ImmutableDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.06372096072114 > 28.678544994103074.
IsChangePoint: Marked as a change because one of 4/5/2023 11:16:13 AM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -30.840691194083934 (T) = (0 -29983.887024653) / Math.Sqrt((58502.37431372103 / (23)) + (101199.03982157727 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.09680082329814221 = (27337.58617584709 - 29983.887024653) / 27337.58617584709 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableHashSet - Duration of single invocation 30.32 μs 32.73 μs 1.08 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<String>.ImmutableHashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.72961513211054 > 32.118814469121496.
IsChangePoint: Marked as a change because one of 4/5/2023 6:15:03 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -17.91965283205727 (T) = (0 -32618.762840887095) / Math.Sqrt((139903.52594900012 / (23)) + (149967.47578337672 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.06743512581233914 = (30558.07519549591 - 32618.762840887095) / 30558.07519549591 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 185.21 μs 219.41 μs 1.18 0.41 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 219.4075287078373 > 195.10829571693026.
IsChangePoint: Marked as a change because one of 4/6/2023 1:55:21 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -58.20746821828837 (T) = (0 -219761.94657888898) / Math.Sqrt((860357.8866407182 / (23)) + (6260099.308236786 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.1812207718516285 = (186046.46296085705 - 219761.94657888898) / 186046.46296085705 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels May 9, 2023
@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3e8f17a65a068fca3d19fa5cd43a7e1cd414a5ae
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchmark.GetChildKeysTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddChainedConfigurationNoDelimiter - Duration of single invocation 614.56 μs 691.19 μs 1.12 0.02 False
AddChainedConfigurationEmpty - Duration of single invocation 10.43 ms 28.76 ms 2.76 0.40 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchmark.GetChildKeysTests*'

Payloads

Baseline
Compare

Histogram

Benchmark.GetChildKeysTests.AddChainedConfigurationNoDelimiter

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 691.1870628881987 > 648.0551650379069.
IsChangePoint: Marked as a change because one of 4/19/2023 1:39:34 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -9.538408082504391 (T) = (0 -669053.6250722164) / Math.Sqrt((194146205.87499458 / (22)) + (208740830.1342201 / (21))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (21) - 2, .025) and -0.06582221373880205 = (627734.7351630442 - 669053.6250722164) / 627734.7351630442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.761574888888887 > 10.922525795416668.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 5/5/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -36.91362847216741 (T) = (0 -28747999.36971001) / Math.Sqrt((1232956503218.2644 / (34)) + (1960290041084.2651 / (10))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (10) - 2, .025) and -1.6237492826160578 = (10956839.344438538 - 28747999.36971001) / 10956839.344438538 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 11.27 μs 12.31 μs 1.09 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.314636443983911 > 11.881346597006926.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -13.79665625480297 (T) = (0 -12617.95000391291) / Math.Sqrt((6061.447485116022 / (23)) + (174431.12459801493 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.11262038947747587 = (11340.750289358553 - 12617.95000391291) / 11340.750289358553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 27.24 μs 29.58 μs 1.09 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.ImmutableDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.5817267845912 > 29.01065883294753.
IsChangePoint: Marked as a change because one of 4/5/2023 6:15:03 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -18.133331424962236 (T) = (0 -29676.86438856725) / Math.Sqrt((88931.83283086984 / (23)) + (199080.3953755346 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.07595179712559556 = (27581.964608311424 - 29676.86438856725) / 27581.964608311424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 7.93 μs 9.53 μs 1.20 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.53326783809959 > 8.427870448664828.
IsChangePoint: Marked as a change because one of 2/25/2023 3:45:56 AM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -40.61725093443036 (T) = (0 -9588.879345410225) / Math.Sqrt((10545.336482174154 / (23)) + (20410.437866570424 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.1907658346753327 = (8052.699419298231 - 9588.879345410225) / 8052.699419298231 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IFooObjIsIFooInterAlia - Duration of single invocation 327.20 μs 375.00 μs 1.15 0.20 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 374.99534365079364 > 342.9615589029948.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -27.208384159897836 (T) = (0 -380677.6481669433) / Math.Sqrt((94391.71942150916 / (23)) + (82173026.93236075 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.16476559400606464 = (326827.68973081565 - 380677.6481669433) / 326827.68973081565 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@cincuranet cincuranet removed the untriaged New issue has not been triaged by the area owner label May 9, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 9, 2023
@cincuranet cincuranet transferred this issue from dotnet/perf-autofiling-issues May 9, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 9, 2023
@cincuranet
Copy link
Contributor

Commit range is c62f69b...2a216ab. Probably PGO...

This was referenced May 9, 2023
Closed
Closed
@AndyAyersMS
Copy link
Member

Likely a PGO update, but we usually don't see perf swings quite this large. So need to see if there's somethimg amiss with the PGO data, or the way the JIT interacts with it...

@AndyAyersMS AndyAyersMS added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed untriaged New issue has not been triaged by the area owner labels May 9, 2023
@ghost
Copy link

ghost commented May 9, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 23.76 ns 34.01 ns 1.43 0.03 False
ParseSpan - Duration of single invocation 23.26 ns 34.00 ns 1.46 0.03 False
Parse - Duration of single invocation 11.86 ns 15.23 ns 1.28 0.09 False
TryParse - Duration of single invocation 12.63 ns 17.11 ns 1.35 0.05 False
Parse - Duration of single invocation 23.17 ns 35.23 ns 1.52 0.03 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.01435809036666 > 25.3743801477778.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -47.04559135093131 (T) = (0 -34.297548225823824) / Math.Sqrt((0.8005490307930864 / (23)) + (0.31126691835768755 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.44004695426090074 = (23.816965220711797 - 34.297548225823824) / 23.816965220711797 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.0025681010384 > 24.611768631477045.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -46.84627991199859 (T) = (0 -34.268259827484265) / Math.Sqrt((0.8428960124932353 / (23)) + (0.27599784156844087 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.43893043423743877 = (23.815091412424486 - 34.268259827484265) / 23.815091412424486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.233663593779918 > 12.463789074068497.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -17.55416495146352 (T) = (0 -15.950257694152228) / Math.Sqrt((0.56611465688463 / (23)) + (0.459656587047386 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.3111793848142055 = (12.164817323155507 - 15.950257694152228) / 12.164817323155507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.TryParse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.110386249177225 > 13.017706441037802.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -21.476102037711637 (T) = (0 -16.274352469310102) / Math.Sqrt((0.20947989456445193 / (23)) + (0.46853402978353403 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.3053286466629373 = (12.467628371533376 - 16.274352469310102) / 12.467628371533376 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.228765884344334 > 24.448686000325765.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -30.66097569867656 (T) = (0 -34.29792558643127) / Math.Sqrt((1.8703417116844718 / (23)) + (0.6842109628104064 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.4320577592674471 = (23.9500993339654 - 34.29792558643127) / 23.9500993339654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 538.46 ns 686.08 ns 1.27 0.00 True
Compare_Same_Upper - Duration of single invocation 1.15 μs 2.28 μs 1.98 0.36 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 686.0821531424299 > 564.3787270447281.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -579.3427571726128 (T) = (0 -685.6942626411378) / Math.Sqrt((0.725414759068659 / (23)) + (0.714626546587564 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.2760794439870314 = (537.3444936145421 - 685.6942626411378) / 537.3444936145421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.27794682493587 > 1.1776218656975188.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -93.48272309886674 (T) = (0 -2250.1618006414624) / Math.Sqrt((2809.1135448678656 / (23)) + (636.5590553633673 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -1.0531923371231457 = (1095.9332742271495 - 2250.1618006414624) / 1095.9332742271495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 1.57 μs 1.67 μs 1.06 0.38 False
HashSet - Duration of single invocation 8.88 μs 9.66 μs 1.09 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.Stack(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6693939692714146 > 1.6459481093363364.
IsChangePoint: Marked as a change because one of 3/2/2023 4:26:55 AM, 4/9/2023 1:58:52 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -16.106220302535547 (T) = (0 -1729.8309426630458) / Math.Sqrt((50.09530921594631 / (23)) + (2164.523623082486 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.10560978515321734 = (1564.5944580920318 - 1729.8309426630458) / 1564.5944580920318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.AddGivenSize<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.659812924644845 > 9.415499566867954.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -14.906857164051546 (T) = (0 -9967.163168512167) / Math.Sqrt((5329.6980621772145 / (23)) + (89724.65834760114 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.11157541090470617 = (8966.699938423375 - 9967.163168512167) / 8966.699938423375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 13.44 ns 16.83 ns 1.25 0.08 False
Parse - Duration of single invocation 13.24 ns 16.24 ns 1.23 0.12 False
Parse - Duration of single invocation 11.03 ns 12.15 ns 1.10 0.29 False
TryParse - Duration of single invocation 14.55 ns 15.74 ns 1.08 0.08 False
TryParse - Duration of single invocation 10.57 ns 12.09 ns 1.14 0.32 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.Parse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.826146143964596 > 14.119530080196824.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 3/13/2023 6:11:08 PM, 4/17/2023 1:54:35 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -46.76185571127924 (T) = (0 -16.518261848600194) / Math.Sqrt((0.02085848296423869 / (23)) + (0.07074501304336984 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.2271602380239204 = (13.460558235817132 - 16.518261848600194) / 13.460558235817132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.23877566364392 > 13.90532514864634.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 3/13/2023 12:04:12 PM, 4/17/2023 6:40:18 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -48.36895243368514 (T) = (0 -16.200777365552934) / Math.Sqrt((0.005068269745208753 / (23)) + (0.07270830373925728 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.22127048370204414 = (13.265511270233459 - 16.200777365552934) / 13.265511270233459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.Parse(value: "0")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.15398815823343 > 10.635719684883263.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -18.676969945448686 (T) = (0 -12.135074364221301) / Math.Sqrt((0.14678046554799734 / (23)) + (0.06144337488222862 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.17437792912437924 = (10.333193483352723 - 12.135074364221301) / 10.333193483352723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.TryParse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.739083277852119 > 15.320294620178725.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -16.26890816159469 (T) = (0 -15.79932298951513) / Math.Sqrt((0.034046559937457296 / (23)) + (0.0682911959451056 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.07623619696671498 = (14.680163177975475 - 15.79932298951513) / 14.680163177975475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt16.TryParse(value: "0")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.085731777599758 > 10.746377702684738.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -8.268523248929515 (T) = (0 -11.909829426312127) / Math.Sqrt((0.5631650447733183 / (23)) + (0.09868101589272804 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.13456361094530517 = (10.49727781802291 - 11.909829426312127) / 10.49727781802291 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt32 - Duration of single invocation 5.68 ns 10.58 ns 1.86 0.04 False
FormatterInt32 - Duration of single invocation 1.55 ns 7.09 ns 4.56 0.17 False
FormatterInt32 - Duration of single invocation 4.45 ns 8.83 ns 1.99 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.575831831229376 > 5.957003043233291.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -1574.8660868034437 (T) = (0 -10.562699249205188) / Math.Sqrt((0.00012342313623343488 / (23)) + (8.982280015006975E-05 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.8622160891109447 = (5.672112549649386 - 10.562699249205188) / 5.672112549649386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.08956926702956 > 1.6324580612478683.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -3087.9906465206955 (T) = (0 -7.088097218182055) / Math.Sqrt((4.623066496265888E-06 / (23)) + (6.317721837668212E-05 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -3.5553670501149837 = (1.555988165213414 - 7.088097218182055) / 1.555988165213414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.830698653458 > 4.6845527687024315.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/25/2023 12:29:35 AM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -118.10554486920714 (T) = (0 -8.822609142330025) / Math.Sqrt((0.03070856775007168 / (23)) + (0.00021011967589717185 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.9645442995820404 = (4.490918908882354 - 8.822609142330025) / 4.490918908882354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 27.24 μs 30.06 μs 1.10 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.ImmutableDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.06372096072114 > 28.678544994103074.
IsChangePoint: Marked as a change because one of 4/5/2023 11:16:13 AM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -30.840691194083934 (T) = (0 -29983.887024653) / Math.Sqrt((58502.37431372103 / (23)) + (101199.03982157727 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.09680082329814221 = (27337.58617584709 - 29983.887024653) / 27337.58617584709 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableHashSet - Duration of single invocation 30.32 μs 32.73 μs 1.08 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<String>.ImmutableHashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.72961513211054 > 32.118814469121496.
IsChangePoint: Marked as a change because one of 4/5/2023 6:15:03 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -17.91965283205727 (T) = (0 -32618.762840887095) / Math.Sqrt((139903.52594900012 / (23)) + (149967.47578337672 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.06743512581233914 = (30558.07519549591 - 32618.762840887095) / 30558.07519549591 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4772b5d2b5facba2bbfcb6b25c4410c1a93d1557
Compare 442141d696ee7d4ae54420974559e99738d753e6
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 185.21 μs 219.41 μs 1.18 0.41 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 219.4075287078373 > 195.10829571693026.
IsChangePoint: Marked as a change because one of 4/6/2023 1:55:21 PM, 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/8/2023 3:52:55 PM falls between 4/29/2023 8:32:21 PM and 5/8/2023 3:52:55 PM.
IsRegressionStdDev: Marked as regression because -58.20746821828837 (T) = (0 -219761.94657888898) / Math.Sqrt((860357.8866407182 / (23)) + (6260099.308236786 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.1812207718516285 = (186046.46296085705 - 219761.94657888898) / 186046.46296085705 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: AndyAyersMS
Labels:

os-linux, arch-x64, area-CodeGen-coreclr, runtime-coreclr, PGO, needs-area-label
Milestone: -

@AndyAyersMS AndyAyersMS removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 9, 2023
@AndyAyersMS AndyAyersMS added this to the 8.0.0 milestone May 9, 2023
@AndyAyersMS
Copy link
Member

AndyAyersMS commented May 10, 2023 •
edited
Loading

Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty

[link]

Looking at the most dramatic slowdown:

image

The big improvement on April 17 seems to fall in the range 223d152...9aefa9d -- the only possible candidate is #84582

@AndyAyersMS
Copy link
Member

Can repro

Method Job Toolchain Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
AddChainedConfigurationEmpty Job-JFZJCV \base-rel\corerun.exe 12.61 ms 0.236 ms 0.221 ms 12.66 ms 12.29 ms 13.03 ms 1.00 0.00 143.24 KB 1.00
AddChainedConfigurationEmpty Job-UJTJCB \diff-rel\corerun.exe 25.80 ms 0.427 ms 0.379 ms 25.90 ms 25.01 ms 26.32 ms 2.05 0.04 143.32 KB 1.00 
;; base 0535b03e15fe8c6751a99ff3f64cf08ae812e99b

50.26%   1.827E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerStyle(value class System.ReadOnlySpan`1<wchar>,value class System.Globalization.NumberStyles,class System.Globalization.NumberFormatInfo,!!0&)
27.68%   1.006E+07   Tier-1   [System.Private.CoreLib]Ordinal.CompareStringIgnoreCase(wchar&,int32,wchar&,int32)
08.25%   3E+06       Tier-1   [System.Private.CoreLib]OrdinalCasing.LastIndexOf(value class System.ReadOnlySpan`1<wchar>,value class System.ReadOnlySpan`1<wchar>)
05.20%   1.89E+06    Tier-1   [System.Private.CoreLib]PackedSpanHelpers.IndexOf(int16&,int16,int32)
01.90%   6.9E+05     native   coreclr.dll
01.07%   3.9E+05     Tier-1   [Microsoft.Extensions.Configuration]ConfigurationKeyComparer.Compare(class System.String,class System.String)
01.02%   3.7E+05     Tier-1   [System.Private.CoreLib]String.GetNonRandomizedHashCodeOrdinalIgnoreCase()

;; diff 52b81afe83ba93ea786708a66abf3985419bbea7

51.29%   3.705E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerStyle(value class System.ReadOnlySpan`1<wchar>,value class System.Globalization.NumberStyles,class System.Globalization.NumberFormatInfo,!!0&)
24.61%   1.778E+07   Tier-1   [System.Private.CoreLib]Number.IsWhite(int32)
13.12%   9.48E+06    Tier-1   [System.Private.CoreLib]Ordinal.CompareStringIgnoreCase(wchar&,int32,wchar&,int32)
04.11%   2.97E+06    Tier-1   [System.Private.CoreLib]OrdinalCasing.LastIndexOf(value class System.ReadOnlySpan`1<wchar>,value class System.ReadOnlySpan`1<wchar>)
02.99%   2.16E+06    Tier-1   [System.Private.CoreLib]PackedSpanHelpers.IndexOf(int16&,int16,int32)

Note this could be stale PGO data -- so might resolve itself with another PGO update. We took one with #85866 and just took another one with #85968.

But let's drill in and see....

@AndyAyersMS
Copy link
Member

AndyAyersMS commented May 10, 2023 •
edited
Loading

In case you're wondering how AddChainedConfigurationEmpty ends up spending basically all of its time parsing integers, it is sorting on strings using a custom comparer that eventually bottoms out here:

runtime/src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationKeyComparer.cs

Lines 70 to 98 in d80ba2f

static int Compare(ReadOnlySpan<char> a, ReadOnlySpan<char> b)
{
#if NETCOREAPP
bool aIsInt = int.TryParse(a, out int value1);
bool bIsInt = int.TryParse(b, out int value2);
#else
bool aIsInt = int.TryParse(a.ToString(), out int value1);
bool bIsInt = int.TryParse(b.ToString(), out int value2);
#endif
int result;
if (!aIsInt && !bIsInt)
{
// Both are strings
result = a.CompareTo(b, StringComparison.OrdinalIgnoreCase);
}
else if (aIsInt && bIsInt)
{
// Both are int
result = value1 - value2;
}
else
{
// Only one of them is int
result = aIsInt ? -1 : 1;
}
return result;
}

So not too surprising that it gets lumped in with parsing benchmarks like we see here.

@AndyAyersMS
Copy link
Member

Comparing jit dumps/disasm shows that the diff version has static pgo. This impacts inlining; some of the calls to IsWhite

;; base

Inlines into 06001907 [via ExtendedDefaultPolicy] System.Number:TryParseBinaryIntegerStyle[int](System.ReadOnlySpan`1[ushort],int,System.Globalization.NumberFormatInfo,byref):int:
  [INL01 IL=0002 TR=000001 06001A8B] [INLINED: callee: below ALWAYS_INLINE size] System.ReadOnlySpan`1[ushort]:get_IsEmpty():bool:this
  [INL02 IL=0030 TR=000762 06001919] [INLINED: call site: profitable inline] System.Number:IsWhite(int):bool
  [INL03 IL=0065 TR=000793 06001919] [INLINED: call site: profitable inline] System.Number:IsWhite(int):bool
...
  [INL65 IL=1067 TR=000119 06001919] [INLINED: call site: profitable inline] System.Number:IsWhite(int):bool
  [INL66 IL=1094 TR=000213 06001919] [INLINED: call site: profitable inline] System.Number:IsWhite(int):bool

;; diff

Inlines into 06001905 [via ExtendedDefaultPolicy] System.Number:TryParseBinaryIntegerStyle[int](System.ReadOnlySpan`1[ushort],int,System.Globalization.NumberFormatInfo,byref):int:
  [INL01 IL=0002 TR=000001 06001A89] [INLINED: callee: below ALWAYS_INLINE size] System.ReadOnlySpan`1[ushort]:get_IsEmpty():bool:this
  [INL02 IL=0030 TR=000798 06001917] [INLINED: call site: profitable inline] System.Number:IsWhite(int):bool
  [INL00 IL=0065 TR=000832 06001917] [FAILED: call site: unprofitable inline] System.Number:IsWhite(int):bool
...
  [INL00 IL=1067 TR=000124 06001917] [FAILED: call site: unprofitable inline] System.Number:IsWhite(int):bool
  [INL00 IL=1094 TR=000221 06001917] [FAILED: call site: unprofitable inline] System.Number:IsWhite(int):bool

So this suggests that the static PGO data is not a good match for this test case, since we see IsWhite prominent in the diff profiles.

IsWhite is called in a loop at the very start of TryParseBinaryIntegerStyle. As you can see from the profile data, we never observe this loop to execute and so the in loop call in BB05 is considered cold and not inlined.

image

@AndyAyersMS
Copy link
Member

This particular test case fills a dictionary with "empty keys" which are just whitespace:

https://github.com/dotnet/performance/blob/d5e733fe3416e9402556ffe5c4a4e54a99b06d38/src/benchmarks/micro/libraries/Microsoft.Extensions.Configuration/GetChildKeysTests.cs#L29

so the implication is that we never try parsing any integers with leading or trailing whitespace allowed and actual leading or trailing whitespace in our training scenarios, and when we rely on this static PGO data we make some decisions that adversely impact this test.

@AndyAyersMS
Copy link
Member

AndyAyersMS commented May 10, 2023 •
edited
Loading

I realized the above is actually looking at perf on Windows and this report was about a regression in Linux. So I pulled Linux (and arm64) data into my data query and surprisingly Linux perf (orange line below) on this test has returned to normal

image

Not sure if this is telling us that Linux PGO data is different, or that the jit responds differently on Linux, or that all the PGO data has been updated but the Windows runs haven't seen it yet for some reason.

At any rate I'm going to look into some of the other regressions here and come back to this one.

@AndyAyersMS
Copy link
Member

AndyAyersMS commented May 10, 2023 •
edited
Loading

System.Tests.Perf_UInt64.TryParse(value: "12345")

[link]

image

This one seems linux-x64 specific.

Actually, if you check out the ampere results there seems to be a regression there too. I think those two may share the same MIBC? So am going to take a look at how that data differs from the windows data.

@AndyAyersMS
Copy link
Member

For the cases above, on windows there is no static PGO for System.Number:TryParseBinaryIntegerStyle[ulong](System.ReadOnlySpan`1[ushort],int,System.Globalization.NumberFormatInfo,byref):int (note this is a different method than in the previous benchmark, that one had [int] as its first generic param).

On Linux there is static PGO, but it is very "thin" -- only 4 calls to the method. This leads to cascading diffs but as far as I can tell the main impact is to block layout and some flow opts; notably inlining is the same.

This lack of coverage for uint parsing likely explains the bulk of the regressions above.

One mitigation we might contemplate is to toss out static PGO data if there's not a lot of coverage in a method. Another might be to try and use profile blend to avoid over-fitting to what is likely a very small and not representative sample. Blend is perhaps not quite ready for general use, but I will try enabling it here and see what it does.

@AndyAyersMS
Copy link
Member

Updated and collated results

Notes Recent Score Orig Score x64-lin-intel Benchmark
1.68 2.76 1.68
2.76		 Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
1.44 4.56 1.44
4.56		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
1.28 1.31 1.28
1.31		 System.Tests.Perf_Int32.TryParseSpan(value: "2147483647")
1.26 1.27 1.26
1.27		 System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
1.24 1.25 1.24
1.25		 System.Tests.Perf_Int32.ParseSpan(value: "2147483647")
1.22 1.26 1.22
1.26		 System.Tests.Perf_Int32.TryParse(value: "2147483647")
1.22 1.24 1.22
1.24		 System.Tests.Perf_Int32.Parse(value: "2147483647")
1.21 1.27 1.21
1.27		 System.Tests.Perf_Int32.Parse(value: "-2147483648")
1.18 1.27 1.18
1.27		 System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648")
1.15 1.18 1.15
1.18		 System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)
1.14 1.15 1.14
1.15		 System.Tests.Perf_Int32.TryParse(value: "-2147483648")
1.12 1.98 1.12
1.98		 System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
1.10 1.23 1.10
1.23		 System.Tests.Perf_UInt16.Parse(value: "12345")
1.09 1.20 1.09
1.20		 System.Collections.ContainsTrue(String).HashSet(Size: 512)
1.09 1.25 1.09
1.25		 System.Tests.Perf_UInt16.Parse(value: "65535")
1.08 1.19 1.08
1.19		 System.Tests.Perf_Int32.TryParse(value: "12345")
1.06 1.15 1.06
1.15		 PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia
1.04 1.10 1.04
1.10		 System.Collections.ContainsKeyFalse(String, String).ImmutableDictionary(Size: 512)
1.03 1.21 1.03
1.21		 System.Tests.Perf_Int32.TryParseSpan(value: "12345")
1.03 1.16 1.03
1.16		 System.Tests.Perf_Int32.Parse(value: "12345")
1.03 1.09 1.03
1.09		 System.Collections.TryGetValueFalse(String, String).ImmutableDictionary(Size: 512)
1.01 1.14 1.01
1.14		 System.Tests.Perf_Int32.TryParseSpan(value: "4")
1.01 1.08 1.01
1.08		 System.Collections.ContainsFalse(String).ImmutableHashSet(Size: 512)
0.99 1.08 0.99
1.08		 System.Tests.Perf_UInt16.TryParse(value: "65535")
0.99 1.28 0.99
1.28		 System.Tests.Perf_UInt64.Parse(value: "12345")
0.99 1.46 0.99
1.46		 System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")
0.98 1.35 0.98
1.35		 System.Tests.Perf_UInt64.TryParse(value: "12345")
0.98 1.43 0.98
1.43		 System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")
0.98 1.86 0.98
1.86		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)
0.98 1.09 0.98
1.09		 System.Collections.AddGivenSize(String).HashSet(Size: 512)
0.97 1.52 0.97
1.52		 System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")
0.97 1.06 0.97
1.06		 System.Collections.AddGivenSize(String).Stack(Size: 512)
0.96 1.12 0.96
1.12		 System.Tests.Perf_Int64.Parse(value: "12345")
0.96 1.20 0.96
1.20		 System.Tests.Perf_Int64.ParseSpan(value: "12345")
0.94 1.13 0.94
1.13		 System.Tests.Perf_Int64.TryParseSpan(value: "12345")
0.93 1.17 0.93
1.17		 System.Tests.Perf_Int32.Parse(value: "4")
0.92 1.13 0.92
1.13		 System.Tests.Perf_Int32.ParseSpan(value: "4")
0.91 1.32 0.91
1.32		 System.Tests.Perf_Int64.Parse(value: "-9223372036854775808")
0.91 1.14 0.91
1.14		 System.Tests.Perf_Int32.TryFormat(value: -2147483648)
0.90 1.18 0.90
1.18		 System.Tests.Perf_Int64.TryParse(value: "-9223372036854775808")
0.90 1.29 0.90
1.29		 System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808")
0.90 1.21 0.90
1.21		 System.Tests.Perf_Int64.ParseSpan(value: "-9223372036854775808")
0.87 1.14 0.87
1.14		 System.Tests.Perf_UInt16.TryParse(value: "0")
0.87 1.09 0.87
1.09		 System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)
0.86 1.22 0.86
1.22		 System.Tests.Perf_Int64.Parse(value: "9223372036854775807")
0.86 1.99 0.86
1.99		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)
0.85 1.10 0.85
1.10		 System.Tests.Perf_UInt16.Parse(value: "0")
0.85 1.17 0.85
1.17		 System.Tests.Perf_Int64.ParseSpan(value: "9223372036854775807")
0.83 1.16 0.83
1.16		 System.Tests.Perf_Int64.TryParse(value: "9223372036854775807")
0.81 1.12 0.81
1.12		 Benchmark.GetChildKeysTests.AddChainedConfigurationNoDelimiter
0.80 1.14 0.80
1.14		 System.Tests.Perf_Int64.TryParseSpan(value: "9223372036854775807")
0.79 1.31 0.79
1.31		 System.Linq.Tests.Perf_Enumerable.ElementAt(input: IList)
0.77 1.09 0.77
1.09		 System.Collections.CtorFromCollection(String).HashSet(Size: 512)
0.63 1.45 0.63
1.45		 System.Linq.Tests.Perf_Enumerable.Count(input: ICollection)
0.45 1.09 0.45
1.09		 System.Linq.Tests.Perf_Enumerable.SelectToList(input: Array)
0.21 1.13 0.21
1.13		 System.Collections.IterateForEach(Int32).ImmutableQueue(Size: 512)

@AndyAyersMS AndyAyersMS added the Priority:2 Work that is important, but not critical for the release label Jun 9, 2023
@AndyAyersMS
Copy link
Member

Looks like many of these are now dramatically improved, eg

newplot - 2023-07-25T102919 888

Let me rerun the classifier and see what's left to investigate.

@AndyAyersMS
Copy link
Member

AndyAyersMS commented Jul 25, 2023 •
edited
Loading

Updated classification

Notes Recent Score Orig Score main_x64_ubuntu%2018.04/ Benchmark
0.7ns slower overall with recent data 2.07 4.56 2.07
4.56		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
since improved 1.32 2.76 1.32
2.76		 Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
stabilized & improved w/dynamic PGO 1.31 1.99 1.31
1.99		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)
complicated, but original regression is gone 1.17 1.18 1.17
1.18		 System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)
stablized w/dynamic pgo; only slightly regressed 1.16 1.86 1.16
1.86		 System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)
1.14 1.24 1.14
1.24		 System.Tests.Perf_Int32.Parse(value: "2147483647")
1.13 1.26 1.13
1.26		 System.Tests.Perf_Int32.TryParse(value: "2147483647")
1.13 1.98 1.13
1.98		 System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
1.09 1.25 1.09
1.25		 System.Tests.Perf_Int32.ParseSpan(value: "2147483647")
1.08 1.15 1.08
1.15		 System.Tests.Perf_Int32.TryParse(value: "-2147483648")
1.08 1.27 1.08
1.27		 System.Tests.Perf_Int32.Parse(value: "-2147483648")
1.07 1.31 1.07
1.31		 System.Tests.Perf_Int32.TryParseSpan(value: "2147483647")
1.07 1.27 1.07
1.27		 System.Tests.Perf_Int32.TryParseSpan(value: "-2147483648")
1.05 1.09 1.05
1.09		 System.Collections.AddGivenSize(String).HashSet(Size: 512)
1.03 1.08 1.03
1.08		 System.Collections.ContainsFalse(String).ImmutableHashSet(Size: 512)
1.00 1.20 1.00
1.20		 System.Collections.ContainsTrue(String).HashSet(Size: 512)
1.00 1.15 1.00
1.15		 PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFooInterAlia
0.99 1.06 0.99
1.06		 System.Collections.AddGivenSize(String).Stack(Size: 512)
0.99 1.10 0.99
1.10		 System.Collections.ContainsKeyFalse(String, String).ImmutableDictionary(Size: 512)
0.99 1.27 0.99
1.27		 System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
0.96 1.09 0.96
1.09		 System.Collections.TryGetValueFalse(String, String).ImmutableDictionary(Size: 512)
0.93 1.28 0.93
1.28		 System.Tests.Perf_UInt64.Parse(value: "12345")
0.93 1.25 0.93
1.25		 System.Tests.Perf_UInt16.Parse(value: "65535")
0.92 1.23 0.92
1.23		 System.Tests.Perf_UInt16.Parse(value: "12345")
0.92 1.32 0.92
1.32		 System.Tests.Perf_Int64.Parse(value: "-9223372036854775808")
0.90 1.21 0.90
1.21		 System.Tests.Perf_Int64.ParseSpan(value: "-9223372036854775808")
0.90 1.21 0.90
1.21		 System.Tests.Perf_Int32.TryParseSpan(value: "12345")
0.87 1.09 0.87
1.09		 System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)
0.87 1.09 0.87
1.09		 System.Collections.CtorFromCollection(String).HashSet(Size: 512)
0.86 1.46 0.86
1.46		 System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")
0.85 1.52 0.85
1.52		 System.Tests.Perf_UInt64.Parse(value: "18446744073709551615")
0.85 1.43 0.85
1.43		 System.Tests.Perf_UInt64.TryParse(value: "18446744073709551615")
0.84 1.20 0.84
1.20		 System.Tests.Perf_Int64.ParseSpan(value: "12345")
0.84 1.19 0.84
1.19		 System.Tests.Perf_Int32.TryParse(value: "12345")
0.82 1.31 0.82
1.31		 System.Linq.Tests.Perf_Enumerable.ElementAt(input: IList)
0.80 1.12 0.80
1.12		 System.Tests.Perf_Int64.Parse(value: "12345")
0.80 1.29 0.80
1.29		 System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808")
0.80 1.13 0.80
1.13		 System.Tests.Perf_Int64.TryParseSpan(value: "12345")
0.80 1.08 0.80
1.08		 System.Tests.Perf_UInt16.TryParse(value: "65535")
0.80 1.16 0.80
1.16		 System.Tests.Perf_Int32.Parse(value: "12345")
0.79 1.35 0.79
1.35		 System.Tests.Perf_UInt64.TryParse(value: "12345")
0.79 1.14 0.79
1.14		 System.Tests.Perf_Int32.TryFormat(value: -2147483648)
0.78 1.18 0.78
1.18		 System.Tests.Perf_Int64.TryParse(value: "-9223372036854775808")
0.75 1.17 0.75
1.17		 System.Tests.Perf_Int64.ParseSpan(value: "9223372036854775807")
0.75 1.22 0.75
1.22		 System.Tests.Perf_Int64.Parse(value: "9223372036854775807")
0.73 1.14 0.73
1.14		 System.Tests.Perf_Int64.TryParseSpan(value: "9223372036854775807")
0.72 1.17 0.72
1.17		 System.Tests.Perf_Int32.Parse(value: "4")
0.71 1.16 0.71
1.16		 System.Tests.Perf_Int64.TryParse(value: "9223372036854775807")
0.70 1.13 0.70
1.13		 System.Tests.Perf_Int32.ParseSpan(value: "4")
0.70 1.12 0.70
1.12		 Benchmark.GetChildKeysTests.AddChainedConfigurationNoDelimiter
0.70 1.14 0.70
1.14		 System.Tests.Perf_Int32.TryParseSpan(value: "4")
0.61 1.45 0.61
1.45		 System.Linq.Tests.Perf_Enumerable.Count(input: ICollection)
0.61 1.14 0.61
1.14		 System.Tests.Perf_UInt16.TryParse(value: "0")
0.57 1.10 0.57
1.10		 System.Tests.Perf_UInt16.Parse(value: "0")
0.54 1.09 0.54
1.09		 System.Linq.Tests.Perf_Enumerable.SelectToList(input: Array)
0.21 1.13 0.21
1.13		 System.Collections.IterateForEach(Int32).ImmutableQueue(Size: 512)

@AndyAyersMS
Copy link
Member

Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty

This one is interesting -- while the report here is for ubuntu x64 on 02-May, the data shows more substantial hits to windows platforms(win-x64-intel highlighted below). However, things have since improved across the board.

image

@AndyAyersMS
Copy link
Member

For the most part the issues here have been resolved (some by enabling Dynamic PGO).

There is still too much sensitivity to static PGO in some places, typically these are tests where a cold inline can improve hot code. Not actionable for .NET 8, but something to keep in mind for the future.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@AndyAyersMS AndyAyersMS

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
2 participants
@cincuranet @AndyAyersMS