[DXGI] OBS seems to incorrectly release references to DXGI objects.

[hsiversson]

Operating System Info

Windows 11

Other OS

Win11/Win10

OBS Studio Version

30.2.2

OBS Studio Version (Other)

Not sure if specific version, but seems like a recent regression

OBS Studio Log URL

no log, crash happens in external application being hooked into

OBS Studio Crash Log URL

No response

Expected Behavior

The expectation is that OBS correctly manages DXGI object lifetimes and releases the references in a way that is opaque for the external software. Applications should not be affected by hooking in a breaking way.

Current Behavior

Whenever a swapchain resource is released from host application, there seems to be references lingering on said swapchain object, preventing creation of a new swapchain object.

In Windows 10 or later you are only allowed to have a single IDXGISwapChain object referencing a particular window handle (HWND) using a DXGI_SWAP_EFFECT_FLIP_* setup. The proper way to reconstruct a swapchain with this setup is to first release the initial swapchain object, make sure that the D3D* backend have properly released the object, and then create the new one.
However, because of the lingering reference in OBS, the swapchain object never gets destructed, and thus prevents the application from ever creating a new swapchain.

Calling any of the IDXGIFactory::CreateSwapChainForXYZ functions without successfully releasing all references to the old swapchain object will cause the function to fail and throw the following error:

DXGI ERROR: IDXGIFactory::CreateSwapChain: Only one flip model swap chain can be associate with an HWND, IWindow, or composition surface at a time. ClearState() and Flush() may need to be called on the D3D11 device context to trigger deferred destruction of old swapchains. [ MISCELLANEOUS ERROR https://github.com//pull/297: ]

I reported a similar issue back in December which seems to have been fixed, and it was verified working in later OBS versions.
#9863

However, the issue seems to have resurfaced but in a slightly different flavor.
IDXGISwapChain::Release returns the correct reference count upon execution, which seems to cause our previous failsafe to not trigger. The swapchain object is seemingly released "properly", however it is still not possible to create a new swapchain object as if references were still kept internally.

Steps to Reproduce

1. Launch an external application that initializes a IDXGISwapChain object.
2. Hook OBS to said application.
3. Have the application try to destruct and create a new IDXGISwapChain object in runtime.
4. Observe that the initial swapchain object never gets properly released, and thus creation of new swapchain objects results in E_ACCESSDENIED with the following message:

DXGI ERROR: IDXGIFactory::CreateSwapChain: Only one flip model swap chain can be associate with an HWND, IWindow, or composition surface at a time. ClearState() and Flush() may need to be called on the D3D11 device context to trigger deferred destruction of old swapchains. [ MISCELLANEOUS ERROR https://github.com//pull/297: ]

See MSDN for more info.

Anything else we should know?

I reported a similar issue back in December which seems to have been fixed, and it was verified working in later OBS versions, but the issue is back.
#9863