How to check with mypy that types are *not* compatible

Question

Imagine I am writing a little Python library with one (nonsense) function:

def takes_a_str(x: str) -> str:
    if x.startswith("."):
        raise RuntimeError("Must not start with '.'")
    return x + ";"

For runtime tests of the functionality, I can check it behaves as expected under both correct conditions (e.g. assert takes_a_str('x')=='x;') and also error conditions (e.g. with pytest.raises(RuntimeError): takes_a_str('.')).

If I want to check that I have not made a mistake with the type hints, I can also perform positive tests: I can create a little test function in a separate file and run mypy or pyright to see that there are no errors:

def check_types() -> None:
    x: str = takes_a_str("")

But I also want to make sure my type hints are not too loose, by checking that some negative cases fail as they ought to:

def should_fail_type_checking() -> None:
    x: dict = takes_a_str("")
    takes_a_str(2)

I can run mypy on this and observe it has errors where I expected, but this is not an automated solution. For example, if I have 20 cases like this, I cannot instantly see that they have all failed, and also may not notice if other errors are nestled amongst them.

Is there a way to ask the type checker to pass, and ONLY pass, where a type conversion does not match? A sort of analogue of pytest.raises() for type checking?

Answer 1

mypy and pyright both support emitting errors when they detect unnecessary error-suppressing comments. You can utilise this to do an equivalent of pytest.raises, failing a check when things are type-safe. The static type-checking options that need to be turned on are:

• mypy: warn_unused_ignores = True / --warn-unused-ignores / strict = True / --strict
• pyright: reportUnnecessaryTypeIgnoreComment (see Pyright: Type Check Diagnostics Settings)

Demonstration (mypy Playground, Pyright Playground):

def should_fail_type_checking() -> None:
    # no errors
    x: dict = takes_a_str("")  # type: ignore[assignment] OR # pyright: ignore[reportAssignmentType]
    takes_a_str(2)  # type: ignore[arg-type] OR # pyright: ignore[reportArgumentType]

def check_types() -> None:
    # Failures with mypy and pyright
    # mypy: Unused "type: ignore" comment  [unused-ignore]
    # pyright: Unnecessary "# pyright: ignore" rule: "reportAssignmentType"
    x: str = takes_a_str("")  # type: ignore[assignment] OR # pyright: ignore[reportAssignmentType]

Notes:

• The exact error-suppression code should be provided, as if you don't provide specific codes, mypy and pyright will suppress all errors on a line; in your context, this would be equivalent of doing pytest.raises(BaseException).
• If you're supporting multiple type checkers (e.g. both mypy and pyright), prefer a mypy diagnostic type: ignore[<mypy error code>] - The comment character sequence # type: ignore... is a Python-typing-compliant code which should be recognised by all type-checkers.

Answer 2

Is there a way to ask the type checker to pass, and ONLY pass, where a type conversion does not match? A sort of analogue of pytest.raises() for type checking?

I don't think that's possible with normal tools, for the same reason that there's no way to ask python.exe to only accept a file if there's a syntax error in a certain function.

The purpose of type checking is to verify that the program is self consistent. It does not check specific instances and gives you a score, that's the job of test suites.

If you really want to ensure that your negative examples always fail type checking, you can always script calls to mypy and check for the reported errors, on a per-file basis. Not as clean as pytest.raises(), but much simpler than having a framework for testing type annotations at runtime.