obs-nvenc: Fix reading uninitialized variable

[norihiro]

Description

Define success = false at the beginning of the function static bool nvenc_check().

Motivation and Context

I found a warning below as commented in the PR 11153.

[ 46%] Building C object plugins/obs-nvenc/CMakeFiles/obs-nvenc.dir/nvenc-helpers.c.o
obs-studio/plugins/obs-nvenc/nvenc-helpers.c: In function ‘nvenc_check’:
obs-studio/plugins/obs-nvenc/nvenc-helpers.c:385:16: warning: ‘success’ may be used uninitialized [-Wmaybe-uninitialized]
  385 |         return success;
      |                ^~~~~~~
obs-studio/plugins/obs-nvenc/nvenc-helpers.c:347:14: note: ‘success’ was declared here
  347 |         bool success = config_get_bool(config, "general", "nvenc_supported");
      |              ^~~~~~~

How Has This Been Tested?

OS: Fedora 39 (GCC 13.3.1)

I ran these commands to hit the error.

ln -s libobs.so.30 lib64/libnvidia-encode.so.1
rm bin/obs-nvenc-test
ln -s /usr/bin/true bin/obs-nvenc-test

Then checked log file.

Log from master branch:

---------------------------------                                                                                                               
Loading module: obs-nvenc.so                                                                                                                    
[NVENC] Seems the NVENC test subprocess crashed. Better there than here I guess.                                                                
---------------------------------                                                                                                               

Log from this branch:

---------------------------------                                                                                                               
Loading module: obs-nvenc.so                                                                                                                    
[NVENC] Seems the NVENC test subprocess crashed. Better there than here I guess.                                                                
NVENC not supported                                                                                                                             
Failed to initialize module 'obs-nvenc.so'                                                                                                      
---------------------------------                                                                                                               

It shows Failed to initialize module 'obs-nvenc.so' so that I confirmed the function returns false.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[derrod]

Weird that MSVC and CI didn't catch this but lgtm

[norihiro]

Weird that MSVC and CI didn't catch this but lgtm

CI also shows the warning.

  /home/runner/work/obs-studio/obs-studio/plugins/obs-nvenc/nvenc-helpers.c:385:16: warning: ‘success’ may be used uninitialized [-Wmaybe-uninitialized]
    385 |         return success;
        |                ^~~~~~~
  /home/runner/work/obs-studio/obs-studio/plugins/obs-nvenc/nvenc-helpers.c:347:14: note: ‘success’ was declared here
    347 |         bool success = config_get_bool(config, "general", "nvenc_supported");
        |              ^~~~~~~

Though, we have -Wno-error=maybe-uninitialized so that CI didn't fail. We can probably raise this warning to error for GCC >= 13.0, except librtmp.

obs-studio/cmake/linux/compilerconfig.cmake

Lines 62 to 66 in e25a8b3

	# * Disable false-positive warning in GCC 12.1.0 and later
	# * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105562
	if(CMAKE_C_COMPILER_VERSION VERSION_GREATER_EQUAL 12.1.0)
	add_compile_options(-Wno-error=maybe-uninitialized)
	endif()

[RytoEX]

LGTM