How to mock a Python function so that it won't be called during import?

Question

I am writing some unit tests (using pytest) for someone else's code which I am not allowed to change or alter in any way. This code has a global variable, that is initialized with a function return outside of any function and it calls a function which (while run locally) raises an error. I cannot share that code, but I've coded a simple file that has the same problem:

def annoying_function():
    '''Does something that generates exception due to some hardcoded cloud stuff'''
    raise ValueError() # Simulate the original function raising error due to no cloud connection


annoying_variable = annoying_function()



def normal_function():
    '''Works fine by itself'''
    return True

And this is my test function:

def test_normal_function(): 
    from app.annoying_file import normal_function

    assert normal_function() == True

Which fails due to ValueError from annoying_function, because it is still called during the module import.

Here's the stack trace:

failed: def test_normal_function():
    
>       from app.annoying_file import normal_function

test\test_annoying_file.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
app\annoying_file.py:6: in <module>
    annoying_variable = annoying_function()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def annoying_function():
        '''Does something that generates exception due to some hardcoded cloud stuff'''
>       raise ValueError()
E       ValueError

app\annoying_file.py:3: ValueError

I have tried mocking this annoying_function like this:

def test_normal_function(mocker):
    mocker.patch("app.annoying_file.annoying_function", return_value="foo")
    from app.annoying_file import normal_function

    assert normal_function() == True

But the result is the same.

Here's the stack trace:

failed: thing = <module 'app' (<_frozen_importlib_external._NamespaceLoader object at 0x00000244A7C72FE0>)>
comp = 'annoying_file', import_path = 'app.annoying_file'

    def _dot_lookup(thing, comp, import_path):
        try:
>           return getattr(thing, comp)
E           AttributeError: module 'app' has no attribute 'annoying_file'

..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1238: AttributeError

During handling of the above exception, another exception occurred:

mocker = <pytest_mock.plugin.MockerFixture object at 0x00000244A7C72380>

    def test_normal_function(mocker):
>       mocker.patch("app.annoying_file.annoying_function", return_value="foo")

test\test_annoying_file.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv\lib\site-packages\pytest_mock\plugin.py:440: in __call__
    return self._start_patch(
.venv\lib\site-packages\pytest_mock\plugin.py:258: in _start_patch
    mocked: MockType = p.start()
..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1585: in start
    result = self.__enter__()
..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1421: in __enter__
    self.target = self.getter()
..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1608: in <lambda>
    getter = lambda: _importer(target)
..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1251: in _importer
    thing = _dot_lookup(thing, comp, import_path)
..\..\..\..\.pyenv\pyenv-win\versions\3.10.5\lib\unittest\mock.py:1240: in _dot_lookup
    __import__(import_path)
app\annoying_file.py:6: in <module>
    annoying_variable = annoying_function()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def annoying_function():
        '''Does something that generates exception due to some hardcoded cloud stuff'''
>       raise ValueError()
E       ValueError

app\annoying_file.py:3: ValueError

Also moving the import statement around doesn't affect my result.

From what I've read this happens because the mocker (I'm using pytest-mock) has to import the file with the function it is mocking, and during the import of this file the line annoying_variable = annoying_function() runs and in result fails the mocking process.

The only way that I've found to make this sort-of work is by mocking the cloud stuff that's causing the error in the original code, but I want to avoid this as my tests kind of stop being unit tests then.

Again, I can't modify or alter the original code. I'll be gratefull for any ideas or advice.

Answer 1

As other commenters already noted, the problem that you attempt to solve hints at bigger issues with the code to be tested, so probably giving an answer to how the specific problem can be solved is actually the wrong thing to do. That said, here is a bit of an unorthodox and messy way to do so. It is based on the following ideas:

1. Adjust the source code of annoying_file.py dynamically before importing it, so that annoying_function() will not be called. Given your code example, we can achieve this, for example, by replacing annoying_variable = annoying_function() with annoying_variable = None in the actual source code.
2. Import the dynamically adjusted module rather than the original module.
3. Test normal_function() in the dynamically adjusted module.

In the following code, I assume that

1. a module called annoying_file.py contains the annoying_function(), annoying_variable, and normal_function() from your question,
2. both annoying_file.py and the module that contains the code below live in the same folder.

from ast import parse, unparse, Assign, Constant
from importlib.abc import SourceLoader
from importlib.util import module_from_spec, spec_from_loader

def patch_annoying_variable_in(module_name: str) -> str:
    """Return patched source code, where `annoying_variable = None`"""
    with open(f"{module_name}.py", mode="r") as f:
        tree = parse(f.read())
        for stmt in tree.body:
            # Assign None to `annoying_variable`
            if (isinstance(stmt, Assign) and len(stmt.targets) == 1
                and stmt.targets[0].id == "annoying_variable"):
                stmt.value = Constant(value=None)
                break
    return unparse(tree)
    
def import_from(module_name: str, source_code: str):
    """Load and return a module that has the given name and holds the given code."""
    # Following  https://stackoverflow.com/questions/62294877/
    class SourceStringLoader(SourceLoader):
        def get_data(self, path): return source_code.encode("utf-8")
        def get_filename(self, fullname): return f"{module_name}.py (patched)"
    loader = SourceStringLoader()
    mod = module_from_spec(spec_from_loader(module_name, loader))
    loader.exec_module(mod)
    return mod
        
def test_normal_function():   
    module_name = "annoying_file"
    patched_code = patch_annoying_variable_in(module_name)
    mod = import_from(module_name, patched_code)
    assert mod.normal_function() == True

The code achieves the following:

• With patch_annoying_variable_in(), the original code of annoying_file is parsed. The assignment to annoying_variable is replaced, so that annoying_function() will not be executed. The resulting adjusted source code is returned.
• With import_from(), the adjusted source code is loaded as a module.
• Finally, test_normal_function() makes use of the previous two functions to test the dynamically adjusted module.