Maximum recursion error while calling a python dataframe

[Wattabak]

A bug occurs while trying to initialise a pandas dataframe in the qgis python console. I stumbled upon it writing a plugin and needed to save data to a dataframe. Everything works fine outside the qgis environment. It occurs in any project and I have no idea what causes this.

How to Reproduce

Qgis version: 3.6, appears same in 3.8
OS: Windows 10
python: 3.7.0
pandas: 0.24.2

Opening python console in any project and trying to access a pandas dataframe results in a recursion error. Paste of the console commands run:

import pandas as pd
df = pd.DataFrame(['321'])
df
Traceback (most recent call last):
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "", line 1, in
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\core\base.py", line 78, in repr
return str(self)
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\core\base.py", line 57, in str
return self.unicode()
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\core\frame.py", line 634, in unicode
line_width=width, show_dimensions=show_dimensions)
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\core\frame.py", line 720, in to_string
line_width=line_width)
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\io\formats\format.py", line 389, in init
self.buf = _expand_user(_stringify_path(buf))
File "C:\PROGRA1\QGIS31.8\apps\Python37\lib\site-packages\pandas\io\common.py", line 149, in _stringify_path
from py.path import local as LocalPath
File "C:/PROGRA1/QGIS31.8/apps/qgis/./python\qgis\utils.py", line 739, in _import
if mod and 'file' in mod.dict:
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 168, in dict
self.__makeattr(name)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 148, in __makeattr
result = importobj(modpath, attrname)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 69, in importobj
module = import(modpath, None, None, ['doc'])
File "C:/PROGRA1/QGIS31.8/apps/qgis/./python\qgis\utils.py", line 737, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_path\svnwc.py", line 9, in
import py
File "C:/PROGRA1/QGIS31.8/apps/qgis/./python\qgis\utils.py", line 739, in _import
if mod and 'file' in mod.dict:
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 168, in dict
self.__makeattr(name)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 148, in __makeattr
result = importobj(modpath, attrname)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 69, in importobj
module = import(modpath, None, None, ['doc'])
File "C:/PROGRA1/QGIS31.8/apps/qgis/./python\qgis\utils.py", line 737, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py__metainfo.py", line 1, in
import py
.............
.............
.............
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 168, in dict
self.__makeattr(name)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 148, in __makeattr
result = importobj(modpath, attrname)
File "C:\Users\v.tabakov\AppData\Roaming\Python\Python37\site-packages\py_vendored_packages\apipkg.py", line 69, in importobj
module = import(modpath, None, None, ['doc'])
File "C:/PROGRA1/QGIS31.8/apps/qgis/./python\qgis\utils.py", line 737, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
RecursionError: maximum recursion depth exceeded

here is a pip freeze information from osgeo4w shell

astroid==2.2.5
atomicwrites==1.3.0
attrs==19.1.0
autopep8==1.4.4
certifi==2018.8.24
chardet==3.0.4
colorama==0.4.1
coverage==4.5.1
cycler==0.10.0
decorator==4.3.0
ExifRead==2.1.2
flake8==3.6.0
future==0.16.0
GDAL==2.4.1
httplib2==0.11.3
idna==2.7
importlib-metadata==0.18
ipython-genutils==0.2.0
isort==4.3.20
Jinja2==2.10
jsonschema==2.6.0
jupyter-core==4.4.0
kiwisolver==1.0.1
lazy-object-proxy==1.4.1
lxml==4.3.3
MarkupSafe==1.0
matplotlib==3.0.0
mccabe==0.6.1
mock==2.0.0
more-itertools==7.0.0
nbformat==4.4.0
networkx==2.2
nose2==0.8.0
numpy==1.16.3
OWSLib==0.17.0
packaging==19.0
pandas==0.24.2
pbr==4.3.0
pep8==1.7.1
Pillow==5.3.0
plotly==3.3.0
pluggy==0.12.0
psycopg2==2.7.5
py==1.8.0
pycodestyle==2.5.0
pyflakes==2.0.0
Pygments==2.2.0
pylint==2.3.1
pyodbc==4.0.24
pyparsing==2.4.0
pypiwin32==223
pyproj==1.9.6
PyQt5==5.11.3
pytest==4.6.3
python-dateutil==2.7.5
pytz==2018.7
pywin32==224
PyYAML==3.13
requests==2.19.1
retrying==1.3.3
scipy==1.1.0
Shapely==1.6.4.post2
simplejson==3.16.0
sip==4.19.14
six==1.12.0
traitlets==4.3.2
typed-ast==1.4.0
urllib3==1.23
wcwidth==0.1.7
wrapt==1.11.2
xlrd==1.1.0
xlwt==1.3.0
zipp==0.5.1

QGIS and OS versions

Версия QGIS
3.8.0-Zanzibar
Ревизия
11aff65
Версия Qt при сборке
5.11.2
Текущая версия Qt
5.11.2
Версия GDAL/OGR при сборке
2.4.1
Текущая версия GDAL/OGR
2.4.1
Версия GEOS при сборке
3.7.2-CAPI-1.11.0
Текущая версия GEOS
3.7.2-CAPI-1.11.0 b55d2125
Версия PostgreSQL
9.2.4
Версия SpatiaLite
4.3.0
Версия QWT
6.1.3
Версия QScintilla2
2.10.8
Compiled against PROJ
5.2.0
Running against PROJ
Rel. 5.2.0, September 15th, 2018
OS Version
Windows 10 (10.0)

I reinstalled both qgis and my local python environments a couple of times already with no impact on the issue, out of ideas about what might be the culprit here. I also tried installing a different pandas version in the shell but it had no effect either

[roya0045]

I'm just guessing but you might be hitting your stack limit somehow. Probably since qgis doesn't have pandas built-in and is fetching it from elsewhere. Your best bet would be to check for other plugins that use pandas and see how they use it. Otherwise if you depend on pandas and qgis being able to find it, your plugin wont work for most users.

[roya0045]

An alternative might be to use providers that are in qgis and that support your inputs. This would prevent having to rely on pandas.

[Wattabak]

Yeah I was able to solve that issue, it turned out almost exactly as you guessed, @roya0045 , creating a pandas object requires a py module, and qgis was trying to access it in my local python environment, ignoring its own packages. When I deleted the local python the error was gone too. I am new to qgis, so I wanted to ask if that is the intended behavior of the qgis environment? It is looking for modules that it cannot find in its own env in the other directories, is this intended or am I missing something? Thanks for the quick feedback

[roya0045]

Yes I assume that this is the entended behaviour but as you can see it has a lot of limitation. What I would encourage you to do is to use numpy (as pandas is a numpy wrapper in good part) or use providers, as you know one provider can support csv, so I assume that you could leverage either of those. Or if pandas is necessary you could add it to your plugin if it is not imported by another plugin.

Those are just suggestion, I'm sure other devs might have better advice. What you can also do is ask on the dev mailing list.

In the meantime I think you can close this ticket to migrate the discussion to a place like the mailing list or gitter (is it gitter? I can't recall).

[Wattabak]

Sure, closing it now, thanks a lot for the info and feedback!