ModuleNotFoundError on relative import in folder plugin

[dgw]

Description

Python 3.12, at least, seems to have trouble with submodules of folder plugins, e.g.:

• .sopel/plugins/
  • fun_test_plugin/
    • __init__.py
    • sample.py

If __init__.py does from .sample import *, loading will fail with ModuleNotFoundError: No module named 'fun_test_plugin.sample'

Reproduction steps

Create a plugin folder in .sopel/plugins/fun_test_plugin/ using the file content here:

# __init__.py
import sopel.plugin

from .sample import *

@sopel.plugin.rule(r'^samplemsg')
def samplemsg(bot, trigger):
    bot.say('sample!')

# sample.py
import sopel.module

@sopel.module.rule(r'submsg')
def submsg(bot, trigger):
    bot.say('submsg!')

For simplicity, you can run sopel-plugins enable -a fun_test_plugin on a test install of Sopel, so it will only load the test plugin.

Expected behavior

Plugins should be able to do this. It seemed to work fine before 8.0, and I suspect it has to do with the switch from imp to importlib and how the manual import is handled now vs. before.

Relevant logs

[2024-09-11 23:05:54,367] sopel.bot            INFO     - Loading plugins...
[2024-09-11 23:05:54,373] sopel.bot            ERROR    - Error loading fun_test_plugin: No module named 'fun_test_plugin.sample'
Traceback (most recent call last):
  File "/workspace/sopel/sopel/bot.py", line 279, in setup_plugins
    plugin_handler.load()
  File "/workspace/sopel/sopel/plugins/handlers.py", line 523, in load
    self._module = self._load()
                   ^^^^^^^^^^^^
  File "/workspace/sopel/sopel/plugins/handlers.py", line 493, in _load
    self.module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspace/.sopel/plugins/fun_test_plugin/__init__.py", line 3, in <module>
    from .sample import *
ModuleNotFoundError: No module named 'fun_test_plugin.sample'

Notes

Brought to us on IRC by @clone2727, at which point I verified it and did a little testing. Thanks for the report!

Seems worth investigating a little more, even if the end verdict is "we can't realistically fix this" (importlib + PYTHONPATH is a scary combo). Even if we don't try to fix The Whole Bug, I did notice that there's an assignment into sys.modules (L490 below) that probably should be moved to after a try..except, so it doesn't insert the new modules before checking if it has a valid loader.

sopel/sopel/plugins/handlers.py

Lines 489 to 493 in 1c6aeb9

	module = importlib.util.module_from_spec(self.module_spec)
	sys.modules[self.name] = module
	if not self.module_spec.loader:
	raise exceptions.PluginError('Could not determine loader for plugin: %s' % self.filename)
	self.module_spec.loader.exec_module(module)

Sopel version

8.0.0

Installation method

pip install

Python version

3.12.3

Operating system

Ubuntu 22.04.3 LTS

IRCd

No response

Relevant plugins

See sample code for fun_test_plugin

[SnoopJ]

Seems worth investigating a little more, even if the end verdict is "we can't realistically fix this"

Even if we can't fix it, it might help to emit the search path in that case to let the user know where we were looking. In this case it may provided a hint of PYTHONPATH pain sooner.

[dgw]

I already found a possible fix: put filename here into a list.

sopel/sopel/plugins/handlers.py

Line 474 in 1c6aeb9

submodule_search_locations=filename,

[2024-09-12 01:52:18,914] sopel.bot            INFO     - Loading plugins...
[2024-09-12 01:52:18,951] sopel.module         WARNING  - <snipped deprecation warning about `sopel.module` import>
[2024-09-12 01:52:18,952] sopel.bot            INFO     - Plugin loaded: fun_test_plugin

ModuleSpec documentation says this is supposed to be set to a list, and the current code putting a bare string there could certainly explain the problem.