Optimize AtomicU128::{load, store}

[ibraheemdev]

Processors that enumerate support for Intel® AVX (by setting the feature flag CPUID.01H:ECX.AVX[bit 28]) guarantee that the 16-byte memory operations performed by the following instructions will always be carried out atomically:

• MOVAPD, MOVAPS, and MOVDQA.
• VMOVAPD, VMOVAPS, and VMOVDQA when encoded with VEX.128.
• VMOVAPD, VMOVAPS, VMOVDQA32, and VMOVDQA64 when encoded with EVEX.128 and k0 (masking disabled).

(Note that these instructions require the linear addresses of their memory operands to be 16-byte aligned.)

AtomicU128::{load, store} can take advantage of this instead of using the more expensive cmpxchg instruction. See this GCC issue/patch for details: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688.

[ibraheemdev]

As of ARM v8.4 the LDP/STP instructions are guaranteed to be single-copy atomic for 16 byte accesses:

Changes to single-copy atomicity in Armv8.4 In addition to the single-copy atomicity requirements listed above:

Instructions that are introduced in FEAT_LRCPC are single-copy atomic when all of the following conditions are true:

• All bytes being accessed are within the same 16-byte quantity aligned to 16 bytes.
• Accesses are to Inner Write-Back, Outer Write-Back Normal cacheable memory.

If FEAT_LSE2 is implemented, all loads and stores are single-copy atomic when all of the following conditions are true:

• Accesses are unaligned to their data size but all bytes being accessed are within a 16-byte quantity that is aligned to 16 bytes.
• Accesses are to Inner Write-Back, Outer Write-Back Normal cacheable memory.

If FEAT_LSE2 is implemented, LDP, LDNP, and STP instructions that load or store two 64-bit registers are single-copy atomic when all of the following conditions are true:

• The overall memory access is aligned to 16 bytes.
• Accesses are to Inner Write-Back, Outer Write-Back Normal cacheable memory.

See also the relevant LLVM patch.

[taiki-e]

Btw, recently I learned that powerpc64 (pwr8+) supports 128-bit atomics (llvm patch. although qemu doesn't seem to support some of them) and added support for it to another library. I plan to do the same with this library.

[taiki-e]

UPDATE: This table is outdated. See the atomic128 module's readme for the latest version.

Once #16 merged, the list of targets that support 128-bit atomics and the instructions used is as follows.

target_arch	load	store	CAS	note
x86_64	cmpxchg16b or vmovdqa	cmpxchg16b or vmovdqa	cmpxchg16b	cmpxchg16b target feature required. vmovdqa requires Intel or AMD CPU with AVX. Both compile-time and run-time detection are supported for cmpxchg16b. vmovdqa is currently run-time detection only. Requires rustc 1.59+ when cmpxchg16b target feature is enabled at compile-time, otherwise requires nightly
aarch64	ldxp/stxp or ldp	ldxp/stxp or stp	ldxp/stxp or casp	casp requires lse target feature, ldp/stp requires lse2 target feature. Both compile-time and run-time detection are supported for lse. lse2 is currently compile-time detection only. Requires rustc 1.59+
powerpc64	lq	stq	lqarx/stqcx.	Little endian or target CPU pwr8+. Requires nightly
s390x	lpq	stpq	cdsg	Requires nightly

Note: Run-time detections require outline-atomics optional feature of this crate EDIT: since 0.3.19, run-time detections are enabled by default.

[taiki-e]

Other optimizations:

• aarch64: Use CASP instead of LDXP/STXP for load if available #20
• x86_64: Detect Intel chips from target-cpu #21
• x86_64: Use byte-wise atomic load for first load in 128-bit RMWs #22

(Apart from these, there have also been some minor optimizations regarding inline assembly since #16 was merged.)

[taiki-e]

AMD is also going to guarantee atomicity of 128-bit SSE: https://gcc.gnu.org/bugzilla//show_bug.cgi?id=104688#c10

We would update the AMD APM manuals in the next revision.

For all AMD architectures,

Processors that support AVX extend the atomicity for cacheable, naturally-aligned single loads or stores from a quadword to a double quadword.

which means all 128b instructions, even the *MOVDQU instructions, are atomic if they end up being naturally aligned.

UPDATE: filed #49