Python doctest dictionary equality test with a strange failure (python bug?)

Question

This test isn't failing appropriately. What is wrong?

This incorrectly passes!?

def drop_keys_conditionally(some_dict):
    """Weird bug where the dictionary equality shouldn't pass but does
    
    >>> d = {'check': '...lala...', 'a': 'a', 'b': 'b', 'key': 'val', 'c': 'c', 'd':'d'}
    >>> drop_keys_conditionally(d)
    {'check': '...lala...', 'key': 'val'}
    """
    invocation_to_correct = 'lala'
    keys_to_drop = [
            "a", 
            # "b", 
            "c", 
            "d"
    ]
    
    if invocation_to_correct in some_dict['check']:
        for k in keys_to_drop:
            some_dict.pop(k)
                
    return some_dict

This correctly fails (see the added comma to force it to fail somehow)

def drop_keys_conditionally(some_dict):
    """Weird bug where the dictionary equality shouldn't pass but does
    
    >>> d = {'check': '...lala...', 'a': 'a', 'b': 'b', 'key': 'val', 'c': 'c', 'd':'d'}
    >>> drop_keys_conditionally(d)
    {'check': '...lala...', 'key': 'val',}
    """
    invocation_to_correct = 'lala'
    keys_to_drop = [
            "a", 
            # "b", 
            "c", 
            "d"
    ]
    
    if invocation_to_correct in some_dict['check']:
        for k in keys_to_drop:
            some_dict.pop(k)
                
    return some_dict

Error message:

002 Weird bug where the dictionary equality shouldn't pass but does
003
004     >>> d = {'check': '...lala...', 'a': 'a', 'b': 'b', 'key': 'val', 'c': 'c', 'd':'d'}
005     >>> drop_keys_conditionally(d)
Expected:
    {'check': '...lala...', 'key': 'val',}
Got:
    {'check': '...lala...', 'b': 'b', 'key': 'val'}

Clearly the 'b':'b' is stil there and should have failed the above version. Why does the first version pass? Python doctest bug?

pytest python_bug.py --doctest-modules -v

Version of python: Python 3.11.1

Further exploration

Perhaps even scarier, calling sorted(the_dict.items()) and doing a doctest against that doesn't work either and also silently passes what should be a simple comparison error. Does only a certain amount of the substring have to be equal for the doctest to pass? That would be crazy. Certainly that's not what's going on, but that seems to be what's going on. I can add in a print statement and clearly see the comparison should not be equal.

Also, if I change the doctest to just be an assert, rather than relying on string comparison, now it works correctly. Wild? The whole point of doctests is that it's already supposed to be doing an assert on the string representation of the output. Buggy? I thought this was what doctests were for, but maybe it's python tribal knowledge python doctests are buggy and don't work correctly?

Answer 1

I'm pretty sure this is because you have doctest.ELLIPSIS turned on:

When specified, an ellipsis marker (...) in the expected output can match any substring in the actual output. This includes substrings that span line boundaries, and empty substrings, so it’s best to keep usage of this simple. Complicated uses can lead to the same kinds of “oops, it matched too much!” surprises that .* is prone to in regular expressions.

The ... in lala... is likely matching a large chunk of output.

I believe it's off by default, but it would exactly explain the behaviour you're seeing.

---

I should say that on Python 3.10.10 (which is what I have installed and easily to hand) your example appears to pass for me with no specific settings set.