Pyright does not detect multiple site-packages directories on Red Hat family of Linux distributions

[born2discover]

Red Hat family of distributions (Fedora et al.) differentiate between Python's purelib and platlib and those are thus set to different directories: <prefix>/lib and <prefix>/lib64 respectively. Python packages are then installed in either of these.

The current Pyright path finding logic does not seem to be aware of the existence of the lib64/python(version)/site-packages directory, and thus any and all modules installed in it can not be resolved by Pyright.

Software versions
Python 3.11.9
Pyright 1.1.369 (VS Code Extension)

Steps de reproduce
On any Red Hat related distribution (this issue is being reported from Fedora 40):

1. Install sqlalchemy version >= 2.0
2. Attempt to import sqlalchemy in any python file.
3. See Import "sqlalchemy" could not be resolved Pyright (reportMissingImports) error.

Observations
I noticed that Pyright already faced the same issue, and #1165 introduced a fix. However, after running Pyright with verboseOutput = true, I noticed the following (emphasis mine):

...
[Info  - 16:46:41] Could not import 'sqlalchemy.orm' in file '/home/eugene/Projects/imexp/app/models/user.py'
[Info  - 16:46:41]   Looking in stubPath 'file:///home/eugene/Projects/imexp/typings'
[Info  - 16:46:41]   Attempting to resolve stub package using root path 'file:///home/eugene/Projects/imexp/typings'
[Info  - 16:46:41]   Attempting to resolve using root path 'file:///home/eugene/Projects/imexp/typings'
[Info  - 16:46:41]   Looking in root directory of execution environment 'file:///home/eugene/Projects/imexp'
[Info  - 16:46:41]   Attempting to resolve stub package using root path 'file:///home/eugene/Projects/imexp'
[Info  - 16:46:41]   Attempting to resolve using root path 'file:///home/eugene/Projects/imexp'
[Info  - 16:46:41]   Looking for typeshed stdlib path
[Info  - 16:46:41]   Looking for typeshed stdlib path
[Info  - 16:46:41]   Attempting to resolve using root path 'file:///(...)/extensions/ms-pyright.pyright-1.1.369-universal/dist/typeshed-fallback/stdlib'
[Info  - 16:46:41]   Typeshed path not found
[Info  - 16:46:41]   Finding python search paths
[Info  - 16:46:41]   Executing interpreter: '/home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11/bin/python'
> [Info  - 16:46:41]   Skipping '/usr/lib/python311.zip' because it is not a valid directory
> [Info  - 16:46:41]   Received 4 paths from interpreter
> [Info  - 16:46:41]     file:///usr/lib/python3.11
> [Info  - 16:46:41]     file:///usr/lib/python3.11/lib-dynload
> [Info  - 16:46:41]     file:///app/lib/python3.11/site-packages
> [Info  - 16:46:41]     file:///home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11/lib/python3.11/site-packages
[Info  - 16:46:41]   Looking in python search path 'file:///usr/lib/python3.11'
...

Which seems to indicate that Pyright does not "see" the lib64 directory. That led me to getPythonSearchPaths and its subsequent call to _getSearchPathResultFromInterpreter. The latter seems to call the python interpreter and retrieve relevant paths by executing a series of commands stored in extractSys variable, which amounts to running the following python code:

import os, os.path, sys
normalize = lambda p: os.path.normcase(os.path.normpath(p))

cwd = normalize(os.getcwd())
orig_sys_path = [p for p in sys.path if p != ""]
sys.path[:] = [p for p in sys.path if p != "" and normalize(p) != cwd]

import sys, json
json.dump(dict(path=orig_sys_path, prefix=sys.prefix), sys.stdout)

Running that on my system gives:

Python 3.11.9 (main, Apr 17 2024, 00:00:00) [GCC 14.0.1 20240411 (Red Hat 14.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, os.path, sys
>>> 
>>> normalize = lambda p: os.path.normcase(os.path.normpath(p))
>>> 
>>> cwd = normalize(os.getcwd())
>>> orig_sys_path = [p for p in sys.path if p != ""]
>>> sys.path[:] = [p for p in sys.path if p != "" and normalize(p) != cwd]
>>> 
>>> import sys, json
>>> json.dump(dict(path=orig_sys_path, prefix=sys.prefix), sys.stdout)
{
  "path": [
    "/usr/lib64/python311.zip", 
    "/usr/lib64/python3.11", 
    "/usr/lib64/python3.11/lib-dynload", 
    "/home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11/lib64/python3.11/site-packages", 
    "/home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11/lib/python3.11/site-packages"
  ], 
  "prefix": "/home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11"
}
>>>

• Notice the presence of the python311.zip in /usr/lib64/.
• However, the log message states "[Info - 16:46:41] Skipping '/usr/lib/python311.zip' because it is not a valid directory" which indicates that between the path retrieval from the python interpreter and the possible returns of the _getSearchPathResultFromInterpreter function, "lib64" is turned into "lib".
• However, that is where I am at a loss. Everything seems to point out to the code in _getSearchPathResultFromInterpreter but I do not see where the issue may lie.
• Finally, recreating the virtual environment and re-installing packages did not habe any impact on the issue. Neither did switching from virtual envs. being created in .cache/ to the in-project .venv/ had no impact on the issue.

Manual solution
Adding lib64 as an extra path in the configuration does seem to resolve the issue:
extraPaths = ['./.venv/lib64/python3.11/site-packages']

[erictraut]

I don't think this is a bug in pyright. It appears to be a problem with your configuration or shell environment.

The log indicates that when pyright shell execs python3 and runs the script to extract the sys.paths, it receives the following paths:

> [Info  - 16:46:41]   Received 4 paths from interpreter
> [Info  - 16:46:41]     file:///usr/lib/python3.11
> [Info  - 16:46:41]     file:///usr/lib/python3.11/lib-dynload
> [Info  - 16:46:41]     file:///app/lib/python3.11/site-packages
> [Info  - 16:46:41]     file:///home/eugene/.cache/pypoetry/virtualenvs/imexp-CA8YYaZW-py3.11/lib/python3.11/site-packages

Those are the paths pyright will use for import resolutions.

That's different from what you're seeing when you manually run the script. That means there's a difference between the shell environment used when pyright is running versus the shell environment that you're using when you run the script manually. You'll need to debug that issue on your system. I'm guessing that there's some environment variable that affects the python interpreter path or its behavior in determining which sys.paths to use.

I'm going to close the issue because I'm pretty confident this isn't a pyright bug, and I don't think there's anything actionable. If you need additional help and can provide more clues, feel free to post them here, and I'll do my best to assist in diagnosing the problem.