Different behaviour between 41.0.0, 41.0.1 and 40.0.2 - PyO3 modules may only be initialized once per interpreter process #9016
Comments
|
Looks like the proximate cause is PyO3/pyo3@78ba70d, which came with fact that cryptography 41 upgrades the version of pyo3 we use. However, that doesn't tell us why zigpy is attempting to initialize the rust module multiple times. Unfortunately, I know nothing about zigpy, so hard to say. |
alex
added
the
waiting-on-reporter
|
This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days. |
|
We found the same issue in ceph mgr repository. When mgr process starts their modules (which are on python), it fails with the same error during pyOpenSsl init: cryptography version is 41.0.1, pyOpenSsl version is 23.2.0. |
I don't think that is an issue on zigpy itself, it is most-likely the embedded python which is simply restarting the python application, might related to https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/guide/src/migration.md#each-pymodule-can-now-only-be-initialized-once-per-process |
|
We upgraded today to 41.0.1 and had to revert it because we found the same problem. In this case the problem was with flask-jwt-extended library: |
|
How are you deploying python such that you're initializing modules multiple times? |
|
from ceph side it's a bit complicated question :) we are running ceph-mgr process where an initialization of submodules happens. These submodules are written on python, probably ceph-mgr initializes cryptography and one of the submodule initializes pyopenssl which re-initializes cryptography for some reason. Current WA is to decrease pyopenssl to March release (23.1.1) and cryptography to 40.0.2. |
|
This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days. |
alex
removed
waiting-on-reporter
|
Is there a simple reproducer we can use to understand more about how this is occurring? Can we make this exception apepar with a single CPython invocation or does it require a more complex runtime with, e.g. subinterpreters. |
|
FYI, I have a django app that uses djangosaml2. It now is broken and shows this error. |
|
Is any other particular mod_wsgi configuration required or present? |
Not that I can see. In case it helps, this is my |
|
Ok, it looks like mod_wsgi uses sub-interpreters by default, https://modwsgi.readthedocs.io/en/latest/user-guides/processes-and-threading.html#python-sub-interpreters it makes sense to me that sub-interpreters would trigger this. Unfortunately I don't think there's anything we can do ourselves here, I think this needs to become an issue on pyo3 to discuss what would be required to support sub-interpreters. |
|
Same issue here, AWS Lambda with Python 3.8 runtime where cryptography 41.0.2 is imported fails to bootstrap. Reverting to 40.0.2 was the solution. |
|
This does not appear to be the same error at all, it has a totally different error message. Please read the first line:
This indicates that your lambda does not have the cffi module installed. |
|
cffi is listed in my poetry.lock and is in Lambda layer, but still I guess if it won't be listed there, version revert won't be solution for me. |
|
This can be fixed with #9016 |
The above comment re. solution seems to link back to this issue itself? |
|
This should be fixed for most use cases with our next release (pyo3 0.20), for actual sub-interpreter use cases, work is needed on pyo3 itself. |
Different behaviour between 41.0.0, 41.0.1 and 40.0.2.
We are using cryptography via zigpy libraries. All of that is embedded into a C++ application which used Embedded Python Libraries in order to run some python codes.
We have detected a behaviour change since 40.0.2, which made our code not working anymore ( see the here after stack trace)
any ideas would be more than welcome
with cryptography 41.0.0 and cryptography 41.0.1
The text was updated successfully, but these errors were encountered: