Static type assertions for testing

[whitphx]

Dear all,

Is there some way to statically test the type inferences?

---

For example, assume that we have a function foo(), whose return type is complex as below.

def foo(type_, is_list):
    if type_ == "str":
        if is_list:
            return ["foo", "bar"]
        else:
            return "foo"
    elif type_ == "int":
        if is_list:
            return [42, 43]
        else:
            return 42
    ...

(It's ridiculous, but just a toy example.)

As a library author, I write type annotations on it to support the library users as below:

@overload
def foo(type_: Literal["str"], is_list: Literal[True]) -> list[str]:
    ...
    
@overload
def foo(type_: Literal["str"], is_list: Literal[False]) -> str:
    ...

So, next I want to write tests to check the type annotations are working as expected like below.

# This is pseudo code
assert return_type_of(foo("str", True)) is list[str]
assert return_type_of(foo("str", False)) is str

To do it, for now, I define a following helper class,

from typing import Generic, TypeVar

T = TypeVar('T')


class is_(Generic[T]):
    def __init__(self, arg: T) -> None:
        pass

use it in the test code as below,

is_[list[str]](foo("str", True))
is_[str](foo("str", False))

and check it with mypy.

My question is, is there any standard/authentic way to do such a thing?

---

FYI, In other words, I want to write tests like the TypeScript DefinitelyTyped, for example this, where there are assertions on the static types

[srittau]

I don't think there is a standard way to do this. In fact, in typeshed issue python/typeshed#1339 has been open for quite a while. While we have other tests in typeshed, we still miss the ability to test complex corner cases directly. But @sobolevn mentioned pytest-mypy-plugins in that thread, maybe this is something to look into?

[sobolevn]

Yes, I am using (and maintaining) pytest-mypy-plugins for all my projects. Some demos:

• https://github.com/typeddjango/django-stubs/tree/master/tests/typecheck
• https://github.com/dry-python/returns/tree/master/typesafety
• https://github.com/dry-python/classes/tree/master/typesafety

I've also covered this in a tutorial: https://sobolevn.me/2019/08/testing-mypy-types

[whitphx]

Thank you so much!
It seems to be exactly what I have to look into.
I'm going to check out pytest-mypy-plugins

[whitphx]

Thank you, but I found pytest-mypy-plugins did not work well with my actual project.

Its features such as parameterization or capability to assert type-check errors look great, but it's too hard for me to bootstrap the typing tests with it.
(And it's bit hard to write the test cases in YAML files, not .py)

Thank you again, and it was nice I could know that there is no standard way to test the static type inferences - then I will start with a my own minimum helper class just for my own case.

---

FYI:
The project is https://github.com/whitphx/streamlit-webrtc, and I'm writing a test case like below:

from typing import Generic, TypeVar, Optional

from streamlit_webrtc import webrtc_streamer, VideoProcessorBase  # My library

T = TypeVar('T')


# My helper class for type assertion
class is_(Generic[T]):
    def __init__(self, arg: T) -> None:
        pass


def test() -> None:
    class VideoProcessor(VideoProcessorBase):
        foo: int = 42

        def recv(self, frame):
            return frame

    ctx = webrtc_streamer(key="foo", video_processor_factory=VideoProcessor)

    # I want to test whether the generic class and its type inference work as expected
    is_[Optional[VideoProcessor]](ctx.video_processor)
    is_[None](ctx.audio_processor)
    if ctx.video_processor:
        is_[int](ctx.video_processor.foo)

The mypy.ini is

[mypy]
python_version = 3.8
ignore_missing_imports = True

I wrote a test case with pytest-mypy-plugins as below,

- case: test
  main: |
    from streamlit_webrtc import webrtc_streamer, VideoProcessorBase

    class VideoProcessor(VideoProcessorBase):
        foo: int = 42

        def recv(self, frame):
            return frame

    ctx = webrtc_streamer(key="foo", video_processor_factory=VideoProcessor)
    reveal_type(ctx.video_processor)  # N: Revealed type is 'VideoProcessor'

but encountered the following massive errors - doesn't it work correctly when the dependencies do not have type definitions?

E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
E   Actual:
E     ../../../../../../path/to/streamlit-webrtc/.venv/lib/python3.8/site-packages/_pytest/_argcomplete:103: error: Cannot find implementation or library stub for module named "argcomplete.completers" (diff)
E     ../../../../../../path/to/streamlit-webrtc/.venv/lib/python3.8/site-packages/_pytest/_argcomplete:103: error: Cannot find implementation or library stub for module named "argcomplete" (diff)
E     ../../../../../../path/to/streamlit-webrtc/.venv/lib/python3.8/site-packages/_pytest/_code/code:33: error: Skipping analyzing "pluggy": module is installed, but missing library stubs or py.typed marker (diff)
E     ../../../../../../path/to/streamlit-webrtc/.venv/lib/python3.8/site-packages/_pytest/_io/terminalwriter:70: error: Cannot find implementation or library stub for module named "colorama" (diff)

(...a lot of lines)

E     ../../../../../../path/to/streamlit-webrtc/streamlit_webrtc/receive:8: error: Skipping analyzing "aiortc.mediastreams": module is installed, but missing library stubs or py.typed marker (diff)
E     ../../../../../../path/to/streamlit-webrtc/streamlit_webrtc/relay:1: error: Skipping analyzing "aiortc.contrib.media": module is installed, but missing library stubs or py.typed marker (diff)
E     ../../../../../../path/to/streamlit-webrtc/streamlit_webrtc/webrtc:16: error: Skipping analyzing "aiortc": module is installed, but missing library stubs or py.typed marker (diff)
E     ../../../../../../path/to/streamlit-webrtc/streamlit_webrtc/webrtc:17: error: Skipping analyzing "aiortc.contrib.media": module is installed, but missing library stubs or py.typed marker (diff)
E     ../../../../../../path/to/streamlit-webrtc/streamlit_webrtc/webrtc:18: error: Skipping analyzing "aiortc.mediastreams": module is installed, but missing library stubs or py.typed marker (diff)

[whitphx]

Thank you, it all makes sense.

I missed the big pitfall you mentioned that unexpected Any cannot be caught with the is_ assertion.
I will try pyright to use this style of assertion: x: Literal["bool"] = reveal_type(True | False).

The 2-arguments version of reveal_type or assert_type is exactly what I want. In my case, I want to use the expected_type assertion.
Though I'm not a specialist in Python typing, I personally support the proposal.

I don't have the preference on the func name (overloading the current reveal_type or not), and JelleZijlstra's opinion also makes sense.
Any of them, assert_type(True | False, bool) too, is perfect for me.

It's truly good if it is released publicly whichever as a standardized API or a specific API of pyright.

[erictraut]

Based on this discussion, I decided to change the mechanism used by pyright. The new release (1.1.211, which I just published) now implements the mechanism proposed above. Its treatment of reveal_type now accepts optional expected_type and expected_text keyword parameters. I've also updated all of pyright's tests cases to use the new mechanism, and I'm pretty happy with it. Let me know if this addresses your use case and whether you have any suggestions.

@JelleZijlstra, I'm open to switching to a different standard function name (like assert_type) if that is the consensus of the typing community.

Here's an example:

[whitphx]

Great! I can use it now and it's perfect for my purpose, thank you!

BTW, regarding the func name, I changed my mind after I actually wrote and executed some tests.
Now I think assert_type is better.
The reason, in addition to JelleZijlstra's, is it prints out the type information to stdout, which is not needed for testing.
reveal_type() is originally designed for printing the types, and I think we should create a different method for a different purpose.

Thank you again!

---

Update:
Now I also have another aspect that reveal_type() is considered as a general-purpose method for debugging and its functionality such as printing or assertion should be controlled via runtime options of the type checker.
I don't know which is better - I again don't have any preferences on it. Both work at least in my case.

[JelleZijlstra]

It did occur to me that we could perhaps use breakpoint() from https://www.python.org/dev/peps/pep-0553/ instead of reveal_type().

[thejcannon]

For those who are chained to mypy, I just published a tiny plugin which does roughly this (and handles the Any case correctly AFAIK)
https://pypi.org/project/mypy-typing-asserts/

Really hope this makes it into typing though so the community can get standard support <3

[rchen152]

For what it's worth, pytype has separate reveal_type and assert_type functions: https://google.github.io/pytype/faq.html#can-i-find-out-what-pytype-thinks-the-type-of-my-expression-is. We added assert_type in response to a user request pretty much identical to the one here - users wanted to be able to assert on types in their tests.

[martindemello]

incidentally, from a dev rather than a user perspective assert_type has been useful in simplifying some of our unit tests, since we could go from

x = InferTypes("code block")
assertTypes(x, "pyi fragment")

to

CheckTypes("code block with some assert_type statements included")

[sobolevn]

that certain code fails type checking

This is a very common problem for mypy plugins. Plugins might have multiple custom failures. For example, we check all of them in our typesafety test cases: https://github.com/dry-python/classes/blob/master/typesafety/test_typeclass/test_validation/test_body.yml#L21-L69

[JelleZijlstra]

It would also be useful for typeshed to be able to test complex signatures. For example, it would be nice if I could commit the test case from python/typeshed#6941 into the repo somehow and ensure that it keeps working with all type checkers.

The type checker-specific error codes are probably overkill though, so let's drop that.

[AlexWaygood]

Wait, yes, I stand corrected -- it is something I have definitely felt a need for.

[whitphx]

It would also be useful for typeshed to be able to test complex signatures.

I totally agree, and I want it so much.
It's being done in the TypeScript's DefinitelyTyped repo like this. There should be the like in Python.

[Viicos]

The type checker-specific error codes are probably overkill though, so let's drop that.

My current workaround is to use assert_type knowing the type is wrong, and add a type: ignore comment. With mypy at least, you can explicitly error on unused type: ignore comments so that isn't accidentally used

[graingert]

For those coming to this discussion mypy now has native support for assert_type https://mypy.readthedocs.io/en/stable/error_code_list.html#check-types-in-assert-type-assert-type

from typing_extensions import assert_type

assert_type([1], list[int])  # OK

assert_type([1], list[str])  # Error

[whitphx]

Thank you!

Also it's available from the standard lib type since 3.11 😄
https://docs.python.org/3/library/typing.html#typing.assert_type

from typing import assert_type

[graingert]

@whitphx can you select this as an answer?

[whitphx]

Oops sorry I forgot it.
Done 👍