Namespace packages are not supported

[flupke]

Goto is tricked by namespace packages and fails.

I made tests to demonstrate this issue in flupke@ad95279

The problem is that imp.find_module() does not support namespace packages.

Should imports.ImportPath._follow_file_system() fiddle with pkg_resources (see http://stackoverflow.com/questions/5741362/alternatives-to-imp-find-module for a beginning of solution)? Or is imp.find_module() something you would like to discard in the future? (maybe by using importlib.import_module(), which does work with namespace packages, but has the disadvantage of actually importing the modules)

[davidhalter]

Seriously, I have absolutely no idea what your namespace packages are. stackoverflow doesn't help me and the PEP which describes this, has been rejected: http://www.python.org/dev/peps/pep-0382/. You (or someone else) really have to help me out here.

Or is imp.find_module() something you would like to discard in the future?

Well, there's a pull request with exactly that idea: #109

It is my intent to get rid of imp and use importlib instead. However, I have absolutely no idea how that should affect "namespace packages", because it looks like they are being created with python code (and Jedi never just executes arbitrary python code).

---

I have just finished writing large parts of the "Jedi Developer" documentation. This may also help:
http://jedi.jedidjah.ch/en/dev/docs/development.html#module-imports

[flupke]

Namespace packages in Python 2 are implemented through setuptools/distribute, since the language does not support them natively: http://packages.python.org/distribute/setuptools.html#namespace-packages

It's used for example in zope.interface, allowing to distribute zope.interface separately from zope.

They are indeed created with python code by putting a special declaration in your top-level __init__.py, that allows distribute to recognize namespace packages and setup special .pth links when you do setup.py develop, or assemble directories in site-packages correctly when you do setup.py install (taking care of not overwriting other packages contributing to the same namespace)

It's the "multiple .pth links pointing to different packages trees" situation that tricks jedi when you have multiple namespace packages installed with setup.py develop.

For example if you have pkg pointing to /src/pkg and pkg.sub pointing to /src/pkg_sub, feeding pkg.sub to goto will start searching in /src/pkg, and fail to find pkg.sub from there because it's in another directories tree.

The fix would be to inspect pkg_resource._namespace_packages and start traversal at the correct location depending if the import path is a namespace package or not.

It's quite a mess, but sadly it's the only "standard" that we have in Python 2.

I'm willing to put some time in developing a patch but first wanted to know your feeling about it.

[davidhalter]

You're now speaking of Python 2, how do namespaces pacakges work in Python 3?

The fix would be to inspect pkg_resource._namespace_packages

how would you get pkg_resource._namespace_packages? Because Jedi works statically and doesn't actually import anything.

[flupke]

Namespace packages in Python 3.3 are very different from what we have in Python 2 (though conceptually similar). They are described extensively in PEP 420. I don't personally use Python 3.3 but from what I understand the main points are they don't require special code in top-level __init__.py anymore, and they are fully supported by the standard library through importlib (this is what #109 is aiming at).

pkg_resources is a part of setuptools/distribute: http://packages.python.org/distribute/pkg_resources.html

[davidhalter]

Since we used #187, instead of #109, we have now Python 3.3 support (through importlib).
Could you try to run your test again (dev branch) and tell me if everything works fine?

[flupke]

I tried my test as-is (with setuptools namespace packages) and it fails.

I also tried to make python 3.3 namespace packages, and it fails too.

[flupke]

To run my tests :

git clone https://github.com/flupke/jedi.git
cd jedi/test
git checkout dev
./test.sh

[flupke]

Note that under python 2.7, it fails in the same way as before (only one of the two sub-packages is found), and neither of the two sub-packages is found under python 3.3.

[davidhalter]

@flupke ./test.sh doesn't exist anymore (we deleted it and use now py.test (see tox config).

However, it's highly possible that this is not working, because I never really checked if namespace packages work and haven't really understood that as well. So I'll be looking into it.

[marc1n]

In our company we are using "namespace packages" in an extensive way. Jedi is great tool, but in our company it has limited usage because of lack of supporting "namespace packages".

I would be greate if someone add support for "namespace packakges".

[davidhalter]

I have read a few things now about namespace packages and I think I know what they are and how I can create them. An easy case are Python3.3 implicit namespace packages, which should already work (with the import internals use of `importlib).

However, namespace packages created with pkgutil and pkg_ressources seem to be much more complicated, especially because they use different patterns on different systems (I didn't quite get how they do it for example on windows, it must be an implementation detail of the above mentioned libraries. The same applies to Linux, there's nothing that always seems to be working).

@flupke @marc1n (or anyone else) Could one of you tell me if this is the only way that namespace packages are being used (1):

---

Let's assume that our namespace package is foo.

Namespace packages are always made up by at least one package in sys.path that defines itself as a namespace package in this (or a similar) way (foo/__init__.py):

try:                                                                            
     import pkg_resources                                                        
     pkg_resources.declare_namespace(__name__)                                   
except ImportError:                                                             
     import pkgutil                                                              
     __path__ = pkgutil.extend_path(__path__, __name__)  

This defines the namespace. If another namespace package is being installed, it will be in a different folder foo, but also in sys.path. This folder can, but must not define __init__.py (2) and may hold 1-n namespace packages (e.g. foo.bar) inside it (3).

Nested namespace packages seem to be possible, but I don't get how they work. Same procedure, with subfolders?

---

(1) If not, please tell me how they could be used differently.
(2) That's at least the difference I saw in @flupke implementation (tests) and the zope packages I played around with.
(3) Not relevant for Jedi, I don't really care.

[flupke]

(1) I don't know if these are the only way, but at least they are the official ways: http://www.python.org/dev/peps/pep-0382/#namespace-packages-today

I never used nested namespaces, and I bet their use is marginal.

[davidhalter]

I never used nested namespaces, and I bet their use is marginal.

Well I could totally see the following use case:
Instead of django.contrib being just a pure internal module it could also be a nested namespace package, where people could add their modules to it.

[davidhalter]

namespace packages should work now, please test (dev branch).

[flupke]

Quickly tested and the goto function works, but omni completion does not.

[flupke]

Hmm now it works... something must have confused jedi-vim (though it happens sometimes too with non-namespace packages so this may be unrelated to this issue).

Great work thanks!

[davidhalter]

Yes, I know what you mean. I'm still trying to figure out those problems.

[sherwoodwang]

I'm using latest jedi-vim. It seems completion for zope.interface doesn't work.

When I use d to find the definition of zope.interface.implementer, it even throws an exception.

Traceback (most recent call last):
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/_compatibility.py", line 24, in find_module_py33
    loader = importlib.find_loader(string)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/importlib/__init__.py", line 70, in find_loader
    raise ValueError('{}.__loader__ is None'.format(name))
ValueError: zope.__loader__ is None
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/username/.vim/bundle/jedi-vim/jedi_vim.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "/Users/username/.vim/bundle/jedi-vim/jedi_vim.py", line 211, in goto
    definitions = [x for x in script.goto_definitions()
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/api/__init__.py", line 364, in goto_definitions
    definitions = resolve_import_paths(definitions)
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/api/__init__.py", line 338, in resolve_import_paths
    scopes.update(resolve_import_paths(set(s.follow())))
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/cache.py", line 41, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/imports.py", line 94, in follow
    types = importer.follow()
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/cache.py", line 41, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/imports.py", line 252, in follow
    return self._do_import(self.import_path, self.sys_path_with_modifications())
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/imports.py", line 281, in _do_import
    bases = self._do_import(import_path[:-1], sys_path)
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/evaluate/imports.py", line 327, in _do_import
    find_module(import_parts[-1])
  File "/Users/username/.vim/bundle/jedi-vim/jedi/jedi/_compatibility.py", line 28, in find_module_py33
    raise ImportError("Originally ValueError: " + e.message)
AttributeError: 'ValueError' object has no attribute 'message'

[davidhalter]

That's a bug, ValueError really doesn't have an attributemessage`. We should probably replace this line with:

ImportError("Originally " + repr(e))

Which looks better anyway. You can PR it, if you want. Also if easily possible add a regression test.

[ghost]

Is there any plan to add support for Namespace packages? Our codebase uses Namespace packages (PEP 420) extensively and as a result jedi pulls up a blank on all of our code.

For the most part, now that PEP 420 works, we now only put init.py into packages that actually have some sort of initialization to perform. Without them we have greater flexibility to compose python packages together.

However, it looks like pkgutil.iter_modules will never return is_pkg=True for Namespace packages. (I'm not sure exactly how the import machinery in Python resolves them, instead, other than NamespacePath and NamespaceLoader are part of it.

[davidhalter]

Is there any plan

No there isn't. Sorry :-) It works like this: Whoever really needs a feature will eventually implement it. I would really like to see this, but currently I don't really have time. I would be happy if you or someone else would take the lead here.

However, it looks like pkgutil.iter_modules will never return is_pkg=True for Namespace packages.

In general, I'm not sure if this is the right way to find that information, because quite frankly I have no clue how Python finds those packages. IMO we should implement it the same way Python implements this anyway, which would mean first investigating how Python does it (even if it's C code, which I don't believe, because it's probably in importlib.py.

[ghost]

Understood. If I were to submit a patch for the documentation for "Unsupported Features", would you prefer namespace packages to be listed in the subsection "Not yet implemented" or "Will probably never be implemented"?

With namespace packages being an integral feature of Python 3, it was a surprise that a code analysis tool of this caliber doesn't support it. (Completely understandably, after reading how the code for jedi handles imports.) A lot of time could be saved if it were in the feature list.

[davidhalter]

It's a question of priorities. I don't have the time to do it right now, but it will definitely be implemented at some point in time. Therefore it's definitely not a "Will probably never be implemented". In general there's a need for people besides me improving Jedi, this could be a good start. :)

[sruggier]

I'm using jedi as part of YouCompleteMe. The codebase I'm working with has a nested set of packages that looks like namespace1.namespace2.regular1.regular2.regular3. The __init__.py in each of the two namespace packages contains code like this:

try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    import pkgutil
    __path__ = pkgutil.extend_path(__path__, __name__)

When I type import namespsace1., the completion list properly includes namespace2 as a proposal, even though it's not the first package in that namespace. However, the completion list is empty forimport namespace1.namespace2.. Finally, if I try to complete on import namespace1.namespace2.regular1., Jedi experiences an internal error, because this snippet of code returns an empty list of paths:

                # It's possible that by giving it always the sys path (and not
                # the __path__ attribute of the parent, we get wrong results
                # and nested namespace packages don't work.  But I'm not sure.
                paths = base.py__path__(sys_path)

Is that related to this issue, or should I file a new one?

[davidhalter]

You probably don't need to create an issue for this, because it's very unlikely that anyone else will tackle it, also it's kind of in this issue.

[davidhalter]

AFAIK Namespace packages are now supported.

[elprans]

AFAIK Namespace packages are now supported.

Support for nested namespace packages is still broken: elprans@ca10e17

[gustavovalverde]

After testing with commit m-novikov@869c72d I can validate that this is not completely working yet.

[davidhalter]

With the help of a few pull requests everything is now working fine. Let me know if there are still issues. It's all on the master branch.

@gustavovalverde It's working perfectly fine. I just checked. You just have to use the right Python version.

[brokenn]

Hi,

I'm failing to get autocompletion working via VSCode/Jedi with a large internal namespace package. I've managed to boil it down to a small example with just Jedi.

I have a namespace package called 'foo', spread across two directories, 'projecta' and 'projectb'

$ tree
.
├── projecta
│   └── foo
│       ├── bar
│       │   └── __init__.py
│       └── __init__.py
└── projectb
    └── foo
        ├── baz
        │   └── __init__.py
        └── __init__.py

The foo/__init__.py file is identical across both directories:

$ cat projecta/foo/__init__.py 
import pkg_resources
pkg_resources.declare_namespace(__name__)

$ cat projectb/foo/__init__.py 
import pkg_resources
pkg_resources.declare_namespace(__name__)

And foo/bar/__init__.py and foo/baz/__init__.py contain a single function each:

$ cat projecta/foo/bar/__init__.py 
def bar():
  return 'bar'

$ cat projectb/foo/baz/__init__.py 
def baz():
  return 'baz'

The namespace packages work in Python as I would expect:

$ PYTHONPATH=projecta:projectb python
>>> import foo; import foo.bar; import foo.baz
>>> foo.bar.bar(), foo.baz.baz()
('bar', 'baz')

But autocompletion with Jedi is only working for one of the two packages, always the first to be listed in the search path. For example, here it will complete for foo.bar.bar() but not foo.baz.baz()

$ PYTHONPATH=projecta:projectb python
>>> source = '''
... import foo; import foo.bar; import foo.baz
... a = foo.bar.b
... b = foo.baz.b
... '''
>>> import jedi
>>> [x.complete for x in jedi.Script(source, 3, 13, '').completions()]
[u'ar']
>>> [x.complete for x in jedi.Script(source, 4, 13, '').completions()]
[]

And with the Python Path swapped around, it will complete for foo.baz.baz() but not foo.bar.bar()

$ PYTHONPATH=projectb:projecta python
>>> source = '''
... import foo; import foo.bar; import foo.baz
... a = foo.bar.b
... b = foo.baz.b
... '''
>>> import jedi
>>> [x.complete for x in jedi.Script(source, 3, 13, '').completions()]
[]
>>> [x.complete for x in jedi.Script(source, 4, 13, '').completions()]
[u'az']
>>> jedi.__version__
'0.12.0'

This is with Jedi 0.12.0

It's entirely possible I've done something wrong and any advice would be appreciated if that is the case. If what I've done is right, I would contend that this bug is not entirely fixed yet. Please let me know if you need any more information.

[davidhalter]

@brokenn Can you please open a new issue? It's probably easier to track. It might not even be related to this issue.

[brokenn]

Sure thing, opened as #1105