[Perf] Linux/x64: 22 Regressions on 2/6/2023 2:23:37 PM

[performanceautofiler]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
MultiplyByScalarBenchmark - Duration of single invocation	23.45 ns	26.27 ns	1.12	0.27	False
MultiplyByScalarOperatorBenchmark - Duration of single invocation	21.84 ns	25.38 ns	1.16	0.33	False
TransformNormalByMatrix4x4Benchmark - Duration of single invocation	125.57 ns	208.56 ns	1.66	0.22	False
TransformByMatrix4x4Benchmark - Duration of single invocation	138.02 ns	243.16 ns	1.76	0.21	False
DistanceSquaredBenchmark - Duration of single invocation	26.60 ns	30.73 ns	1.16	0.20	False
AddFunctionBenchmark - Duration of single invocation	18.43 ns	20.32 ns	1.10	0.25	False
ReflectBenchmark - Duration of single invocation	46.88 ns	73.86 ns	1.58	0.19	False
DotBenchmark - Duration of single invocation	13.88 ns	14.99 ns	1.08	0.32	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector3.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.27033689844652 > 23.152625026134427.
IsChangePoint: Marked as a change because one of 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -9.908033364824423 (T) = (0 -25.51348386733012) / Math.Sqrt((1.5045727601803833 / (55)) + (0.2900665267403583 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.13989621596944665 = (22.3822866589935 - 25.51348386733012) / 22.3822866589935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.MultiplyByScalarOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.384233110132886 > 22.957418814201088.
IsChangePoint: Marked as a change because one of 1/12/2023 10:41:19 PM, 1/13/2023 4:37:08 PM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -7.488403492637522 (T) = (0 -25.653168556967334) / Math.Sqrt((1.0393657498532596 / (55)) + (0.7120591652433594 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.14881031139255624 = (22.33020395322816 - 25.653168556967334) / 22.33020395322816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.TransformNormalByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.56005735237468 > 131.43785193279797.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -72.86897270112517 (T) = (0 -206.6141127680182) / Math.Sqrt((8.540344149094482 / (33)) + (24.499559545837545 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.6300409665853177 = (126.7539387067324 - 206.6141127680182) / 126.7539387067324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.TransformByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 243.16258065004908 > 140.0499358060909.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -51.75462273365964 (T) = (0 -225.49461172099984) / Math.Sqrt((7.881307163305325 / (33)) + (77.31960212877073 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.6988946008596081 = (132.7301950379405 - 225.49461172099984) / 132.7301950379405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.730706661123953 > 28.180712298556298.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 1/6/2023 5:32:27 AM, 1/26/2023 10:17:57 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -17.05971948982348 (T) = (0 -30.920151096328652) / Math.Sqrt((0.8697660097010909 / (55)) + (0.12511628247053522 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.13601703090285958 = (27.21803481392756 - 30.920151096328652) / 27.21803481392756 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.AddFunctionBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.32230574221454 > 19.423872130199072.
IsChangePoint: Marked as a change because one of 12/12/2022 1:34:22 PM, 12/14/2022 5:20:21 PM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -6.613397682185492 (T) = (0 -20.677096981523636) / Math.Sqrt((1.1138295810421868 / (55)) + (0.24484148742867218 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.10045827226509264 = (18.789532963356805 - 20.677096981523636) / 18.789532963356805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.ReflectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.85740757368214 > 49.12633889100288.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -24.511894863097115 (T) = (0 -66.8809113004009) / Math.Sqrt((0.5700378551225441 / (33)) + (16.5498238233046 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.42115292474123117 = (47.06102357885153 - 66.8809113004009) / 47.06102357885153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.DotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.989399861251968 > 14.456191669035562.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -2.5739642274355887 (T) = (0 -16.46653032921173) / Math.Sqrt((0.7191876560627429 / (55)) + (3.4435799261581788 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.17114287812224435 = (14.060223254410591 - 16.46653032921173) / 14.060223254410591 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in System.Numerics.Tests.Perf_Vector4

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DistanceBenchmark - Duration of single invocation	35.84 ns	39.28 ns	1.10	0.22	False
AddOperatorBenchmark - Duration of single invocation	22.23 ns	26.36 ns	1.19	0.31	False
DistanceSquaredBenchmark - Duration of single invocation	32.94 ns	36.23 ns	1.10	0.21	False
MultiplyByScalarOperatorBenchmark - Duration of single invocation	26.91 ns	32.05 ns	1.19	0.29	False
AddFunctionBenchmark - Duration of single invocation	22.07 ns	24.81 ns	1.12	0.27	False
TransformVector3ByMatrix4x4Benchmark - Duration of single invocation	154.37 ns	238.64 ns	1.55	0.17	False
MultiplyFunctionBenchmark - Duration of single invocation	21.98 ns	25.62 ns	1.17	0.37	False
MultiplyByScalarBenchmark - Duration of single invocation	27.72 ns	37.38 ns	1.35	0.31	False
MultiplyOperatorBenchmark - Duration of single invocation	22.43 ns	24.81 ns	1.11	0.31	False
TransformByMatrix4x4Benchmark - Duration of single invocation	154.39 ns	243.92 ns	1.58	0.15	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector4.DistanceBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.27794406282074 > 37.46623534072858.
IsChangePoint: Marked as a change because one of 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -13.634665249583863 (T) = (0 -39.386226971732626) / Math.Sqrt((0.9214265569228877 / (55)) + (0.23237435161304398 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.10461598198469149 = (35.65603577540712 - 39.386226971732626) / 35.65603577540712 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.361161250651005 > 24.529656367711215.
IsChangePoint: Marked as a change because one of 12/12/2022 1:34:22 PM, 12/14/2022 9:46:59 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -5.133739745752217 (T) = (0 -25.337983682560278) / Math.Sqrt((4.845226226800399 / (55)) + (0.4760742589314814 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.10157328195376002 = (23.00163239037589 - 25.337983682560278) / 23.00163239037589 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.DistanceSquaredBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.234151119372214 > 34.49577610477196.
IsChangePoint: Marked as a change because one of 1/22/2023 2:59:30 AM, 1/22/2023 4:26:49 PM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -8.747872091022499 (T) = (0 -36.599561886176346) / Math.Sqrt((2.7308086608973667 / (55)) + (0.2662355747725243 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.08870729735361413 = (33.61744885438087 - 36.599561886176346) / 33.61744885438087 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.MultiplyByScalarOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.05427025002726 > 28.39135631609412.
IsChangePoint: Marked as a change because one of 12/14/2022 11:14:18 PM, 2/4/2023 10:21:07 PM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -3.3781068989416805 (T) = (0 -30.511105193327047) / Math.Sqrt((1.9679486039961354 / (49)) + (6.820476923752505 / (10))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (10) - 2, .025) and -0.10386339192125904 = (27.64029083365369 - 30.511105193327047) / 27.64029083365369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.AddFunctionBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.80588291871051 > 23.160770919668835.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/12/2022 1:34:22 PM, 12/14/2022 5:20:21 PM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -25.359894365341784 (T) = (0 -24.91363329614047) / Math.Sqrt((0.401551333444531 / (55)) + (0.020018129843785693 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.1272902541474872 = (22.100460111740606 - 24.91363329614047) / 22.100460111740606 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformVector3ByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 238.63823822351156 > 162.82016457281654.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -30.338624220377646 (T) = (0 -224.46361919696832) / Math.Sqrt((42.29195382095276 / (33)) + (92.39025019915118 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.42288184878117674 = (157.75281650352144 - 224.46361919696832) / 157.75281650352144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.MultiplyFunctionBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.624165353902523 > 23.432758733611998.
IsChangePoint: Marked as a change because one of 12/6/2022 8:34:17 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -13.098988145784206 (T) = (0 -25.341306393220346) / Math.Sqrt((1.3028554934659378 / (55)) + (0.09374045867286669 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.12639102069976915 = (22.497788003918114 - 25.341306393220346) / 22.497788003918114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.MultiplyByScalarBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.38081356512988 > 28.665919662314256.
IsChangePoint: Marked as a change because one of 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -3.354410006546536 (T) = (0 -34.20019636832417) / Math.Sqrt((9.007479238400409 / (55)) + (14.291775446923339 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.233954730287196 = (27.715924684177246 - 34.20019636832417) / 27.715924684177246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.813504262268793 > 23.183998143079638.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -11.6321270407222 (T) = (0 -25.294718901218744) / Math.Sqrt((1.2618246880209112 / (55)) + (0.1656001539279136 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.1320516494682715 = (22.344138549773554 - 25.294718901218744) / 22.344138549773554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 243.91577317121462 > 162.61701813089593.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -32.01207243171384 (T) = (0 -226.66347801347203) / Math.Sqrt((9.74644877510658 / (33)) + (121.07692836216066 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.4583403808047526 = (155.42563382109248 - 226.66347801347203) / 155.42563382109248 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in System.Memory.Span<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfAnyThreeValues - Duration of single invocation	123.50 ns	133.25 ns	1.08	0.09	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Char>*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 33)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 133.24545855644106 > 130.85473845662432.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 1/13/2023 4:18:46 AM, 1/27/2023 11:36:27 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -8.536905131837688 (T) = (0 -134.98770368583132) / Math.Sqrt((4.159164575276171 / (55)) + (4.168344891429975 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.07165125209362458 = (125.96234402014038 - 134.98770368583132) / 125.96234402014038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in Benchstone.BenchI.TreeInsert

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test - Duration of single invocation	360.04 μs	392.70 μs	1.09	0.10	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.TreeInsert*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.TreeInsert.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 392.69930725446426 > 378.28222233864886.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/14/2022 5:20:21 PM, 12/17/2022 10:44:50 PM, 1/5/2023 9:29:47 PM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -7.346570949849723 (T) = (0 -388055.0099104735) / Math.Sqrt((114854583.53172863 / (55)) + (27439102.396169078 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.060030757858885744 = (366079.00953203515 - 388055.0099104735) / 366079.00953203515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in System.Tests.Perf_Boolean

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryParse - Duration of single invocation	78.31 ns	83.01 ns	1.06	0.22	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Boolean*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Boolean.TryParse(value: "0")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.01155474685724 > 74.84313411675333.
IsChangePoint: Marked as a change because one of 12/2/2022 5:21:30 AM, 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -4.796022505968534 (T) = (0 -88.59244570118035) / Math.Sqrt((10.513288231228383 / (55)) + (37.18897495205872 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.20012833152146675 = (73.81914364846881 - 88.59244570118035) / 73.81914364846881 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	58719ec90b3bbae527dd81685bf8670b993fe8f9
Compare	876a97d7d652b1c688e0b72e229bc38779609724
Diff	Diff

Regressions in System.Numerics.Tests.Perf_Quaternion

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
InverseBenchmark - Duration of single invocation	66.89 ns	73.73 ns	1.10	0.19	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.InverseBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.73351953196148 > 63.48059733566336.
IsChangePoint: Marked as a change because one of 2/6/2023 11:23:26 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -21.741285090201067 (T) = (0 -74.67628039389867) / Math.Sqrt((7.551501864818913 / (55)) + (0.5717696539916151 / (4))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (55) + (4) - 2, .025) and -0.18220598850684977 = (63.16689402683227 - 74.67628039389867) / 63.16689402683227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[naricc]

@tannergooding Looks like we are seeing regressions from this on the interpreter too : dotnet/runtime@90401dd

@BrzVlad Is there any particular reason that change would be problematic for the interpreter and is there any on going work that would improve it?

[tannergooding]

I'm surprised for these methods given the change in question didn't touch the managed implementation of something like Vector4.DistanceSquared, it merely added the [Intrinsic] attribute so that Mono and RyuJIT could recognize it as intrinsic.

[BrzVlad]

It seems to me that this regression is caused by reliance on vectorized operations. For example in Vector4.Transform, instead of explicit computation of each element, the code is changed to make use of vector operations. I suspect this is what makes it slower, probably the new code would need to check somewhere if Vector.IsHardwareAccelerated

[tannergooding]

It seems to me that this regression is caused by reliance on vectorized operations.

Yes these APIs are marketed as being SIMD accelerated and are not expected to be used when SIMD acceleration is not available (e.g. when IsHardwareAccelerated returns false), the software fallbacks primarily existing for testing purposes and to support scenarios such as the debugger where the speed difference is irrelevant. As such they do not necessarily provide "performant" software fallbacks but instead do the optimal thing for the primary scenario (which is also the simplest and easiest to maintain).

I suspect this is what makes it slower, probably the new code would need to check somewhere if Vector.IsHardwareAccelerated

This would regress things more for the primary scenario since it bloats the IL size and makes the methods more difficult to inline/optimize. This is why it is not done today.

The "fix" is to finish onboarding SIMD support in Mono, which exists on practically every device (only embedded and other customized scenarios not supporting it).

The open PR that adds SIMD support to the interpreter is a good step in this direction, but for this scenario in particular it would require more support around float operations.

Noting that while Vector4.Transform was touched and therefore some perf change is to be understood; that doesn't explain why Vector4.DistanceSquared regressed. That function nor any of the functions it calls directly or indirectly were modified by the PR, they simply had the [Intrinsic] attribute added to them.

[BrzVlad]

I tested Vector4.DistanceSquared locally and there is no difference in performance, nor in the code that is being run. Probably just some random benchmark variation. That's why I tested that other benchmark, since it had the biggest regression, making it very likely to be true.

I think Vector4 and Vector<T> arithmetic operations can be intrinsified with the same exact methods as a Vector128<T>, right ? So we could then easily add them in the interpreter anyway ?

[tannergooding]

I think Vector4 and Vector arithmetic operations can be intrinsified with the same exact methods as a Vector128, right ? So we could then easily add them in the interpreter anyway ?

Vector<T> is effectively identical to Vector128<T> yes. There are a couple differences in whether something is an instance of extension method, and maybe in a couple names, but they should generally be 1-to-1 -- with the note that if the underlying platform supports Vector256<T> (that is Vector256.IsHardwareAccelerated is true) then Vector<T> is Vector256<T> instead.

For Vector4 it is generally the same as Vector128<float>. It is non-generic and has a few other differences/considerations such as having publicly exposed fields (and therefore not being an "opaque simd vector type") that need to be additionally handled. Likewise there are a few functions unique to it, such as DistanceSquared which ideally also get acceleration.

Plane and Quaternion are much the same as Vector4 for the methods marked [Intrinsic]. There are a couple unique functions to these as well, such Quaternion.Conjugate and Plane has the consideration that it is a Vector3 + float, rather than 4x float.

Vector2/Vector3 also get SIMD acceleration but have some other considerations. On all platforms Vector2 is 8-bytes and Vector3 is 12-bytes, but they get "enregistered" where possible and so Vector3 is still internally operated on using a Vector128<float>, just masking off the unused element for operations where that matters (such as Dot). Vector2 is much the same way, but on platforms where Vector64.IsHardwareAccelerated (such as Arm64) it is directly handled as that (instead of being enregistered using the larger Vector128<float> like on x64).

[kotlarmilos]

@BrzVlad the regressions are mitigated by dotnet/runtime#87822.