JIT: Enabled embedded broadcast for binary ops #87946

Ruihan-Yin · 2023-06-22T22:51:04Z

This PR provides the embedded broadcast support for binary ops, to be more specific, ops using the genHWIntrinsic_R_R_RM path.

Including the following ops:
and, andn, or, xor,
min, max,
div, mul, mull, sub,
variable shiftleftlogical/rightarithmetic/rightlogical

ghost · 2023-06-22T22:51:14Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

This PR provides the embedded broadcast support for binary ops, to be more specific, ops using the genHWIntrinsic_R_R_RM path.

Including the following ops:
and, andn, or, xor,
min, max,
div, mul, mull, sub,
variable shiftleftlogical/rightarithmetic/rightlogical

Author:	Ruihan-Yin
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

Ruihan-Yin · 2023-06-28T18:56:43Z

Hi @tannergooding , the PR should be ready for review, would you please take a look at it?
it mostly includes:

some updates on the metadata in the related tables to enable embedded broadcast in more instructions.
some workaround on the bitwise instructions to make it work correctly with embedded broadcast when the input data type is in 64-bit.
a bug fix where some broadcast nodes with data type less than 32 bits are accidentally contained, made some changes in the contain check to filter out these situations.

src/coreclr/jit/instr.cpp

Ruihan-Yin · 2023-06-29T20:23:53Z

windows x64

Diffs are based on 1,627,004 contexts (467,427 MinOpts, 1,159,577 FullOpts).

MISSED contexts: 2,227 (0.14%)

Overall (+2,406 bytes)

Collection	Base size (bytes)	Diff size (bytes)
aspnet.run.windows.x64.checked.mch	44,062,516	+156
benchmarks.run.windows.x64.checked.mch	7,244,558	+152
benchmarks.run_pgo.windows.x64.checked.mch	26,963,808	+143
benchmarks.run_tiered.windows.x64.checked.mch	10,507,436	+157
coreclr_tests.run.windows.x64.checked.mch	385,009,583	+736
libraries.pmi.windows.x64.checked.mch	59,506,418	+190
libraries_tests.pmi.windows.x64.checked.mch	121,311,207	+220
realworld.run.windows.x64.checked.mch	13,650,175	+652

MinOpts (+383 bytes)

Collection	Base size (bytes)	Diff size (bytes)
benchmarks.run_pgo.windows.x64.checked.mch	10,473,053	+73
benchmarks.run_tiered.windows.x64.checked.mch	7,567,856	+73
coreclr_tests.run.windows.x64.checked.mch	269,424,110	+237

FullOpts (+2,023 bytes)

Collection	Base size (bytes)	Diff size (bytes)
aspnet.run.windows.x64.checked.mch	29,278,684	+156
benchmarks.run.windows.x64.checked.mch	6,870,267	+152
benchmarks.run_pgo.windows.x64.checked.mch	16,490,755	+70
benchmarks.run_tiered.windows.x64.checked.mch	2,939,580	+84
coreclr_tests.run.windows.x64.checked.mch	115,585,473	+499
libraries.pmi.windows.x64.checked.mch	57,987,107	+190
libraries_tests.pmi.windows.x64.checked.mch	114,958,091	+220
realworld.run.windows.x64.checked.mch	12,500,933	+652

Ruihan-Yin · 2023-06-29T20:27:27Z

Fails should be irrelevant.

We had regression of about 2,400 bytes in the code size introduced by the changes. This should be expected as the regression is the conversion from VEX encoding to EVEX encoding when enabling embedded broadcast.
And there is no influence on the throughput.

Do we accept the results?

tannergooding · 2023-07-06T15:48:36Z

Do we accept the results?

The results look good/within reason to me. There are two important bits to call out...

First this change, in the typical case, trades a slight increase in code size (typically +2-bytes) for a decrease in data size (going from sizeof(VectorXXX<T>) to sizeof(T)) thereby increasing data locality and minimizing impact to the cache. This will often be a performance win.

Second, SPMI diffs are not currently tracking/including the data size allocations as part of "bytes of code" metric. HardwareIntrinsics.RayTracer.Packet256Tracer:Shade for example is +2 bytes of code, but -32 bytes of data so its actually a net win of -30 bytes of allocations

tannergooding · 2023-07-06T15:55:54Z

src/coreclr/jit/lowerxarch.cpp

@@ -8138,11 +8138,18 @@ bool Lowering::IsContainableHWIntrinsicOp(GenTreeHWIntrinsic* parentNode, GenTre
        case NI_AVX2_BroadcastScalarToVector256:
        case NI_AVX512F_BroadcastScalarToVector512:
        {
+            var_types baseType = hwintrinsic->GetSimdBaseType();
+            if (baseType == TYP_BYTE || baseType == TYP_UBYTE || baseType == TYP_SHORT || baseType == TYP_USHORT)


This is better handled as

Suggested change

if (baseType == TYP_BYTE || baseType == TYP_UBYTE || baseType == TYP_SHORT || baseType == TYP_USHORT)

if (varTypeIsSmall(baseType))

tannergooding · 2023-07-06T15:56:18Z

CC. @dotnet/jit-contrib for secondary review.

BruceForstall · 2023-07-06T23:44:35Z

superpmi-replay and superpmi-diffs pipelines are failing with messages like:

[17:40:21] ERROR: Couldn't load base metrics summary created by child process
[17:40:21] General fatal error

I wonder if the JIT is crashing (not asserting) with this PR?

Ruihan-Yin · 2023-07-06T23:48:28Z

I wonder if the JIT is crashing (not asserting) with this PR?

First time seeing this fail in this PR, the recent changes might not cause the crashing, I will rebase the PR and re-run the CI again to confirm.

BruceForstall · 2023-07-06T23:53:37Z

src/coreclr/jit/hwintrinsic.h

@@ -598,6 +598,23 @@ struct HWIntrinsicInfo
 #endif
    }

+#if defined(TARGET_XARCH)
+    static bool IsBitwiseOperation(instruction ins)


This seems like an odd location for this function. All the other functions here take a NamedIntrinsic. Should this instead be in emitxarch.h/cpp like IsMovInstruction or HasKMaskRegisterDest ?

moved the function into emitxarch.h/cpp as a static function.

BruceForstall · 2023-07-07T00:08:14Z

src/coreclr/jit/instrsxarch.h

@@ -399,7 +399,6 @@ INST3(aesenclast,       "aesenclast",       IUM_WR, BAD_CODE,     BAD_CODE,
 INST3(aesimc,           "aesimc",           IUM_WR, BAD_CODE,     BAD_CODE,     SSE38(0xDB),                             INS_TT_NONE,                                          REX_WIG      | Encoding_VEX)                                                                                                                                                   // Perform the AES InvMixColumn Transformation
 INST3(aeskeygenassist,  "aeskeygenassist",  IUM_WR, BAD_CODE,     BAD_CODE,     SSE3A(0xDF),                             INS_TT_NONE,                                          REX_WIG      | Encoding_VEX)                                                                                                                                                   // AES Round Key Generation Assist
 INST3(pclmulqdq,        "pclmulqdq" ,       IUM_WR, BAD_CODE,     BAD_CODE,     SSE3A(0x44),                             INS_TT_NONE,                         Input_64Bit    | REX_WIG      | Encoding_VEX                   | INS_Flags_IsDstDstSrcAVXInstruction)                                                                                           // Perform a carry-less multiplication of two quadwords
-


nit: no need to delete this empty line?

tannergooding · 2023-07-07T00:27:18Z

I wonder if the JIT is crashing (not asserting) with this PR?

There are similar failures in multiple PRs from what I've seen recently.

Noting that SPMI diffs is faulting in Python when processing the diffs summary:

Traceback (most recent call last):
  File "C:\h\w\B2080989\p\superpmi.py", line 4667, in <module>
    sys.exit(main(args))
  File "C:\h\w\B2080989\p\superpmi.py", line 4558, in main
    success = asm_diffs.replay_with_asm_diffs()
  File "C:\h\w\B2080989\p\superpmi.py", line 2019, in replay_with_asm_diffs
    self.write_asmdiffs_markdown_summary(write_fh, asm_diffs)
  File "C:\h\w\B2080989\p\superpmi.py", line 2170, in write_asmdiffs_markdown_summary
    write_row(*t)
  File "C:\h\w\B2080989\p\superpmi.py", line 2165, in write_row
    num_missed_base / num_contexts * 100,
ZeroDivisionError: division by zero

Replay is similarly failing with things like:

[17:42:50] Invoking: C:\h\w\B6DA095E\p\superpmi.exe -v ewi -r C:\h\w\B6DA095E\t\tmpdsiibugo\repro -p -jitoption JitStressRegs=0x80 -f C:\h\w\B6DA095E\t\tmpdsiibugo\coreclr_tests.run.windows.x64.checked.mch_fail.mcl -metricsSummary C:\h\w\B6DA095E\t\tmpdsiibugo\coreclr_tests.run.windows.x64.checked.mch_metrics.csv C:\h\w\B6DA095E\p\clrjit_win_x64_x64.dll C:\h\w\B6DA095E\p\artifacts\spmi\mch\02e334af-4e6e-4a68-9feb-308d3d2661bc.windows.x64\coreclr_tests.run.windows.x64.checked.mch
[17:42:50] ERROR: Couldn't load base metrics summary created by child process
[17:42:50] ERROR: Couldn't load base metrics summary created by child process
[17:42:50] ERROR: Couldn't load base metrics summary created by child process
[17:42:50] ERROR: Couldn't load base metrics summary created by child process
[17:42:50] General fatal error
[17:42:50] Running SuperPMI replay of C:\h\w\B6DA095E\p\artifacts\spmi\mch\02e334af-4e6e-4a68-9feb-308d3d2661bc.windows.x64\libraries.crossgen2.windows.x64.checked.mch

Did we recently get a JIT/EE version change and potentially not have up to date baselines?

BruceForstall · 2023-07-07T00:48:00Z

Noting that SPMI diffs is faulting in Python when processing the diffs summary:

superpmi.py should gracefully handle that. But I think that's a symptom that superpmi.exe crashed.

Did we recently get a JIT/EE version change and potentially not have up to date baselines?

If the GUID changed, we wouldn't have this problem because we would only pick up matched MCH files.

If someone changed the interface without changing the GUID, then anything is possible.

BruceForstall · 2023-07-07T00:50:12Z

I kicked off runs on main (not on a PR) so we can see if there are failures in the main branch:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=331428&view=results
https://dev.azure.com/dnceng-public/public/_build/results?buildId=331429&view=results

BruceForstall · 2023-07-07T18:38:58Z

@Ruihan-Yin The superpmi pipelines are now running clean. I would suggest to merge up to HEAD and re-push your PR to trigger re-run of these pipelines.

Ruihan-Yin · 2023-07-07T18:44:37Z

The superpmi pipelines are now running clean. I would suggest to merge up to HEAD and re-push your PR to trigger re-run of these pipelines.

Thanks for the notification, will resolve the reviews and rebase the branch soon.

and, andn, or, xor, min, max, div, mul, mull, sub, variable shiftleftlogical/rightarithmetic/rightlogical

JIT used to use a uniform intrinsic for bitwise operations with all data types, embedded broadcast is sensitive to input size in this case, adding a helper to let emitter aware when input size is long/ulong.

when embedded broadcast is actually enabled

There are cases when broadcast node are falsely contained by a embedded broadcast compatible node, while the data type is actually not supported Adding extra logics to avoid this situation.

instructions with either long or ulong as basetype should be reset to qword instructions.

make the typecheck based on broadcast node it self.

use `varTypeIsSmall` type check to cover all the unsupported data type in embedded broadcast.

1. put the IsBitwiseInstruction to a proper place. 2. nit: restored unnecessary line delete.

Ruihan-Yin · 2023-07-10T16:51:59Z

Hi @BruceForstall, some tests are cancelled for some reasons, I suppose it is not relevant to the changes in this PR, shall I re-run the test again? And can you please review the new changes, if any. Thanks!

EgorBo · 2023-07-14T20:10:17Z

src/coreclr/jit/instr.cpp

@@ -1230,6 +1230,30 @@ void CodeGen::inst_RV_RV_TT(
    if (IsEmbBroadcast)
    {
        instOptions = INS_OPTS_EVEX_b;
+        if (emitter::IsBitwiseInstruction(ins) && varTypeIsLong(op2->AsHWIntrinsic()->GetSimdBaseType()))


nit: do we need IsBitwiseInstruction at all since we have a switch over ins anyway? (we can just remove the unreached in debug

Possibly not. I'll let Ruihan-Yin fix this in a follow up PR if they want to, that way we can land the important bit before the snap.

Thanks for the feedback, we will fix this point in the following PR on embedded broadcast.

dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 22, 2023

ghost added the community-contribution Indicates that the PR has been added by a community member label Jun 22, 2023

build-analysis bot mentioned this pull request Jun 23, 2023

LibraryImportGenerator.Unit.Tests crashed in CI #87951

Closed

Ruihan-Yin changed the title ~~Enabled embedded broadcast for binary ops~~ JIT: Enabled embedded broadcast for binary ops Jun 23, 2023

Ruihan-Yin closed this Jun 23, 2023

Ruihan-Yin reopened this Jun 23, 2023

Ruihan-Yin force-pushed the EbBinary branch from ed867d8 to 23d2a5a Compare June 27, 2023 00:10

tannergooding added the avx512 Related to the AVX-512 architecture label Jun 27, 2023

Ruihan-Yin marked this pull request as ready for review June 27, 2023 22:01

build-analysis bot mentioned this pull request Jun 29, 2023

windows-x86 release MSBuild_CMake is failing #88172

Closed

tannergooding reviewed Jun 29, 2023

View reviewed changes

src/coreclr/jit/instr.cpp Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Jun 29, 2023

Tracking issue for CI build timeouts #76454

Closed

tannergooding reviewed Jul 6, 2023

View reviewed changes

tannergooding approved these changes Jul 6, 2023

View reviewed changes

BruceForstall reviewed Jul 7, 2023

View reviewed changes

Ruihan-Yin added 2 commits July 7, 2023 11:52

Enabled embedded broadcast for the following ops:

5ed01d7

and, andn, or, xor, min, max, div, mul, mull, sub, variable shiftleftlogical/rightarithmetic/rightlogical

Bug fix:

0f746c4

JIT used to use a uniform intrinsic for bitwise operations with all data types, embedded broadcast is sensitive to input size in this case, adding a helper to let emitter aware when input size is long/ulong.

Ruihan-Yin added 7 commits July 7, 2023 11:52

reset the instruction in the later phase

7825a18

when embedded broadcast is actually enabled

filter irrelevant data type in embedded broadcast

1513e55

There are cases when broadcast node are falsely contained by a embedded broadcast compatible node, while the data type is actually not supported Adding extra logics to avoid this situation.

update the condition on instruction reset:

a326e58

instructions with either long or ulong as basetype should be reset to qword instructions.

Apply format patch

4468b9f

Resolve reviews:

89120d2

make the typecheck based on broadcast node it self.

Resolve review:

f31da65

use `varTypeIsSmall` type check to cover all the unsupported data type in embedded broadcast.

Resolve reviews:

23c3f6e

1. put the IsBitwiseInstruction to a proper place. 2. nit: restored unnecessary line delete.

Ruihan-Yin force-pushed the EbBinary branch from 3f2a95c to 23c3f6e Compare July 7, 2023 18:55

Ruihan-Yin requested a review from BruceForstall July 13, 2023 17:31

EgorBo reviewed Jul 14, 2023

View reviewed changes

EgorBo approved these changes Jul 14, 2023

View reviewed changes

tannergooding merged commit ef9a07c into dotnet:main Jul 14, 2023
132 of 135 checks passed

JulieLeeMSFT mentioned this pull request Aug 2, 2023

What's new in .NET 8 Preview 7 [WIP] dotnet/core#8438

Closed

3 tasks

BruceForstall mentioned this pull request Aug 3, 2023

Support breakpoints on AVX-512 instructions #89705

Merged

ghost locked as resolved and limited conversation to collaborators Aug 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JIT: Enabled embedded broadcast for binary ops #87946

JIT: Enabled embedded broadcast for binary ops #87946

Ruihan-Yin commented Jun 22, 2023

ghost commented Jun 22, 2023

Ruihan-Yin commented Jun 28, 2023 •

edited

Loading

Ruihan-Yin commented Jun 29, 2023

Ruihan-Yin commented Jun 29, 2023 •

edited

Loading

tannergooding commented Jul 6, 2023

tannergooding Jul 6, 2023

tannergooding commented Jul 6, 2023

BruceForstall commented Jul 6, 2023

Ruihan-Yin commented Jul 6, 2023

BruceForstall Jul 6, 2023

Ruihan-Yin Jul 10, 2023

BruceForstall Jul 7, 2023

tannergooding commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

Ruihan-Yin commented Jul 7, 2023 •

edited

Loading

Ruihan-Yin commented Jul 10, 2023

EgorBo Jul 14, 2023

tannergooding Jul 14, 2023

Ruihan-Yin Jul 14, 2023

	if (baseType == TYP_BYTE \|\| baseType == TYP_UBYTE \|\| baseType == TYP_SHORT \|\| baseType == TYP_USHORT)
	if (varTypeIsSmall(baseType))

JIT: Enabled embedded broadcast for binary ops #87946

JIT: Enabled embedded broadcast for binary ops #87946

Conversation

Ruihan-Yin commented Jun 22, 2023

ghost commented Jun 22, 2023

Ruihan-Yin commented Jun 28, 2023 • edited Loading

Ruihan-Yin commented Jun 29, 2023

windows x64

Ruihan-Yin commented Jun 29, 2023 • edited Loading

tannergooding commented Jul 6, 2023

tannergooding Jul 6, 2023

Choose a reason for hiding this comment

tannergooding commented Jul 6, 2023

BruceForstall commented Jul 6, 2023

Ruihan-Yin commented Jul 6, 2023

BruceForstall Jul 6, 2023

Choose a reason for hiding this comment

Ruihan-Yin Jul 10, 2023

Choose a reason for hiding this comment

BruceForstall Jul 7, 2023

Choose a reason for hiding this comment

tannergooding commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

BruceForstall commented Jul 7, 2023

Ruihan-Yin commented Jul 7, 2023 • edited Loading

Ruihan-Yin commented Jul 10, 2023

EgorBo Jul 14, 2023

Choose a reason for hiding this comment

tannergooding Jul 14, 2023

Choose a reason for hiding this comment

Ruihan-Yin Jul 14, 2023

Choose a reason for hiding this comment

Ruihan-Yin commented Jun 28, 2023 •

edited

Loading

Ruihan-Yin commented Jun 29, 2023 •

edited

Loading

Ruihan-Yin commented Jul 7, 2023 •

edited

Loading