Common: Use Mach VM routines for memory mapping

[stenzek]

Description of Changes

MacOS does not support an equivalent of MAP_FIXED_NOREPLACE via mmap(), which means that our usage for allocating PCSX2's memory map is not thread-safe.

#11282 fixed this for Linux, this PR implements it for MacOS.

I'll follow up later with a signal handler tidy-up, that way we can skip compiling everything in the Linux/ directory on MacOS.

Rationale behind Changes

Preventing a race condition that can cause memory corruption in PCSX2 on startup, while it allocates its memory map.

Suggested Testing Steps

Make sure Mac builds still boot.

@TellowKrinkle I've tagged you for review in case you know anything about these system calls. Apple's documentation is all blank pages, I ended up using the published XNU source as a reference...

[TellowKrinkle]

Not super familiar with mach's memory management APIs but it seems reasonable (untested)

One part I do have a question about is the part that's just getting moved around, but since pthread_jit_write_protect_np is per-thread while s_code_write_depth is static, do/will we have something in place to make sure no two threads attempt to write code at the same time (e.g. ee and sw renderer)?
(Don't have to address it here/now, since it won't affect anyone on x86) Edit: I'm apparently blind

[stenzek]

All good :) I've tested on Sonoma (M2 Pro) and it seems fine.

The only potential issue is mmap() vs mach_vm_deallocate() on unmap, but that only affects the Apple Silicon path, I'll resolve that later.

[atarixle]

Sorry to say that, but this does not work on my 2014 MacBook Pro 13". It runs with Sonoma and Open Core Legacy Patcher. I do not know if this is an issue with Open Core Legacy Patcher, BUT PCSX2-v1.7.5849 introduces this feature, while PCSX2-v1.7.5847 runs flawlessly.
Anybody with an unpatched Mac please test this for me! Btw, does PCSX2 still work on Big Sur (this is the last OS that supports my MacBook Pro 2014 natively).

[CatGreen90]

Latest (1.7.5953) still works for me on a 15" MBP from 2014 running Big Sur.

[atarixle]

So, in Big Sur it works (your observation), in OCLP/Sonoma it doesn't (my observation). Anybody able to try if it works in vanilla Sonoma (Intel)?

[DigitalMajestic]

2019 MBP (Intel), macOS 14.4 (Sonoma), 1.7.5955 works for me.

[atarixle]

I guess as I run OCLP/Sonoma my system is out of support in this case and this error is irrelevant on regular systems.

[TellowKrinkle]

We had another user with a similar issue on a Ryzen hackintosh, check what SIP flags you have enabled, apparently some of them break this.

[atarixle]

here is what my Mac's OCLP/Sonoma state is like:

$ csrutil status
System Integrity Protection status: unknown (Custom Configuration).

Configuration:
  Apple Internal: disabled
  Kext Signing: disabled
  Filesystem Protections: disabled
  Debugging Restrictions: enabled
  DTrace Restrictions: enabled
  NVRAM Protections: enabled
  BaseSystem Verification: enabled

This is an unsupported configuration, likely to break in the future and leave your machine in an unknown state.

[jimjamyaha]

I also have this problem. Running 14.6.1 Sonoma, OCLP nightly build 1.6.0.

PCSX2 only works on version v1.7.5847.

SIP settings are

[TellowKrinkle]

@jimjamyaha Can you check if #11734 helps?

[atarixle]

@TellowKrinkle #11734 works on my MacBook Pro 2014 Sonoma 14.6.1 OCLP.

I just saw that you can set the SIP bits as you want before root-patching. What are these bits set like on a supported Mac with Sonoma?