Allow any version of Python (including 3.11) in metadata

[astrojuanlu]

See https://iscinumpy.dev/post/bound-version-constraints/#pinning-the-python-version-is-special, and discussion in https://discuss.python.org/t/requires-python-upper-limits/12663?u=astrojuanlu, as well as practical problems with leaving the version cap on in #2270 (comment).

Deliberately not adding anything to the changelog, and not touching our CI systems either.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

We've decided to wait a bit to hear from @merelcht and @idanov

[astrojuanlu]

Prompted by @idanov, I added a warning that by default is treated like an error:

❯ kedro info
Traceback (most recent call last):
  File "/Users/juan_cano/.micromamba/envs/kedro311-dev/bin/kedro", line 5, in <module>
    from kedro.framework.cli import main
  File "/Users/juan_cano/Projects/QuantumBlack Labs/kedro/kedro/__init__.py", line 20, in <module>
    warnings.warn(
kedro.KedroPythonVersionWarning: Kedro is not yet fully compatible with this Python version.
To proceed at your own risk and ignore this warning,
run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
or set the PYTHONWARNINGS environment variable accordingly.

And can be treated as a normal warning or ignored in two different ways (see https://docs.python.org/3/library/warnings.html#describing-warning-filters):

❯ python -W "default:Kedro is not yet fully compatible" -m kedro info
/Users/juan_cano/Projects/QuantumBlack Labs/kedro/kedro/__init__.py:20: KedroPythonVersionWarning: Kedro is not yet fully compatible with this Python version.
To proceed at your own risk and ignore this warning,
run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
or set the PYTHONWARNINGS environment variable accordingly.
  warnings.warn(

 _            _
| | _____  __| |_ __ ___
| |/ / _ \/ _` | '__/ _ \
|   <  __/ (_| | | | (_) |
|_|\_\___|\__,_|_|  \___/
v0.18.10

Kedro is a Python framework for
creating reproducible, maintainable
and modular data science code.

No plugins installed
❯ PYTHONWARNINGS="ignore:Kedro is not yet fully compatible" kedro info

 _            _
| | _____  __| |_ __ ___
| |/ / _ \/ _` | '__/ _ \
|   <  __/ (_| | | | (_) |
|_|\_\___|\__,_|_|  \___/
v0.18.10

Kedro is a Python framework for
creating reproducible, maintainable
and modular data science code.

No plugins installed

(custom categories can't be specified, see python/cpython#66733)

[merelcht]

LGTM! 👍

[noklam]

Do we need to merge this in #2417? Not sure how we can test this without adding python 3.11 first, as the test will never be triggered in the CI and basically do nothing for 3.7 - 3.10.

[astrojuanlu]

Merging this into gh-2417 doesn't really solve the bootstrapping problem (that there has to be a 3.11-enabled version to unblock the CI).

On the other hand, the plan is to test these changes - it's just that we'll have to mock the sys.version_info check, rather than using a real Python 3.11 environment.

[noklam]

good point👍

[noklam]

@astrojuanlu It turns out to be tricky with sys modules, I haven't found a way to handle version switch nicely.

A few challenges here

1. To use importlib.reload, it takes a module, so you need to import the module beforehand (which we don't if we expect the test fail). So I use a setup_environment fixture to bypass it and reset the PYTHONWARNING, which I think is fine.
2. mocking sys directly will cause error because this interfere with pytest, the correct way to do it is always mock the local module which is kedro.sys. This is fine, but when we do importlib.reload(kedro) we effectively remove kedro.sys so the mock that we insert is removed.

Any idea how it could work?

[astrojuanlu]

Ugh, I see what you mean. I'll give it some thought.

[astrojuanlu]

The key is to not use reload:

In [1]: import importlib

In [2]: import kedro

In [3]: kedro.__loader__.exec_module(kedro)

In [4]: from unittest.mock import patch

In [5]: with patch("kedro.sys.version_info", (3, 11)):
   ...:     kedro.__loader__.exec_module(kedro)
   ...: 
---------------------------------------------------------------------------
KedroPythonVersionWarning                 Traceback (most recent call last)
Cell In[5], line 2
      1 with patch("kedro.sys.version_info", (3, 11)):
----> 2     kedro.__loader__.exec_module(kedro)

File <frozen importlib._bootstrap_external>:883, in exec_module(self, module)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File ~/Projects/QuantumBlack Labs/kedro/kedro/__init__.py:22
     19     warnings.simplefilter("error", KedroPythonVersionWarning)
     21 if sys.version_info >= (3, 11):
---> 22     warnings.warn(
     23         """Kedro is not yet fully compatible with this Python version.
     24 To proceed at your own risk and ignore this warning,
     25 run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
     26 or set the PYTHONWARNINGS environment variable accordingly.""",
     27         KedroPythonVersionWarning,
     28     )

KedroPythonVersionWarning: Kedro is not yet fully compatible with this Python version.
To proceed at your own risk and ignore this warning,
run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
or set the PYTHONWARNINGS environment variable accordingly.

[SajidAlamQB]

This PR looks great to me! 🚀 I think changing it to a warning when using unsupported Python versions is a good approach.

Thank you @astrojuanlu!

[astrojuanlu]

Thanks a bunch @SajidAlamQB ! This is going in

[noklam]

🚀🚀🚀🚀