Complex higher-order decorator annotations lead to Mypy "Cannot determine type" error

[gandhis1]

I have a draft annotations PR here: https://github.com/holoviz/param/pull/959/files

This is a tricky one, and I can't tell if I'm doing something wrong or if this is a bug.

There are multiple decorators here:

• depends takes a func and several extra parameters. It is not valid as a decorator by itself because of those extra parameters.
• However, when combined with accept_arguments on top of it, it becomes a second-order decorator (takes parameters and returns a first-order decorator).

This is further complicated by the fact that not all parameter combinations of depends are valid, and so I use @overload to only support valid combinations. I have found some old Mypy PRs that confirm that combining a decorator with @overload is supported.

Here's what running mypy on the depends.py file produces:

param/depends.py:146: error: Item "str" of "Parameter | str" has no attribute "owner"  [union-attr]
param/depends.py:148: error: Item "str" of "Parameter | str" has no attribute "owner"  [union-attr]
param/depends.py:148: error: Item "str" of "Parameter | str" has no attribute "name"  [union-attr]
param/depends.py:159: error: Cannot determine type of "depends"  [has-type]

I'm kind of stumped here, I've been staring at this for a bit and I don't see anything wrong.

[Daverball]

I think the order of the decorators might be what's tripping mypy up here, @overload should generally be the outermost decorator. It probably would be easier to just write the actual resulting signature for the overloads, rather than reuse the decorator, it's better documentation too.

Also if I may suggest some simplifications, you are overusing ParamSpec in depends. CallableT should suffice here. Also your return type for the function is incorrect, you wrote what the return type would be after the decorator has already been applied.

The following looks more correct to me:

@overload
def depends(
    *dependencies: str, watch: bool = ..., on_init: bool = ...
) -> Callable[[CallableT], DependsFunc[CallableT]]:
    ...

@overload
def depends(
    *dependencies: Parameter, watch: bool = ..., on_init: bool = ..., **kw: Parameter
) -> Callable[[CallableT], DependsFunc[CallableT]]:
    ...

@accept_arguments
def depends(
    func: CallableT, *dependencies: Dependency, watch: bool = False, on_init: bool = False, **kw: Parameter
) -> DependsFunc[CallableT]:

[Daverball]

The other possibility is that mypy dislikes a Protocol with __call__: CallableT, in which case you'd actually want to switch to ParamSpec. Although it's worth noting that this usually doesn't work very well when combined with methods and classmethods. It's one of the reasons why lru_cache doesn't properly preserve the signature of the function currently.

So it might be better to pretend this internal attribute doesn't exist, return CallableT instead of DependsFunc[CallableT] and ignore the type error where the attribute is being accessed. Or better yet, write a little utility function, that encapsulates the type messiness at a small cost of type safety:

def get_dependency_info(f: Callable[..., Any]) -> DependencyInfo:
   return f._dinfo  # type:ignore[attr-defined]

[gandhis1]

Also if I may suggest some simplifications, you are overusing ParamSpec in depends. CallableT should suffice here.

Good point.

It probably would be easier to just write the actual resulting signature for the overloads, rather than reuse the decorator, it's better documentation too.

This seems to solve the problem, although it feels a little inconsistent (in most contexts you annotate the function and let the decorator do the signature transformation).

Thanks - I have a path forward now.

[Daverball]

FWIW reversing the order of decorators and fixing the return type would've probably worked too, but I think in this particular case it's easier for users to understand if they see the rewritten overloads, rather than ones with an extra decorator that they first have to read and understand.