Can a Python program produce varying outputs from the same input

Question

I have an extremely large math calculation for a python cgi art project that I am working on. The python code runs for days loading database info then making thousands of floating point multiplications and divisions iteratively. It is producing different output with no random functions or other changing inputs. I am using pandas databases. Is it ever possible that a CPU error or some other issue creates math output variance on such a large scale? I went over it many times and can't see why the same logic math and input does not produce the same result every time it is run. Can floating point math ever produce different results? Can cpus produce math errors or variance?

Answer 1

Short answer: Yes, it's theoretically possible.

Real answer: But that's almost certainly not what's happening here (even for huge datasets, the odds of such an issue on any given run are tiny, so if you've run three times and the output has differed every time, either it's not cosmic ray bit flips, or you're a walking break in probability and should definitely go play the lottery). You have a bug, or you've introduced randomness or varying input data that you are unaware of.

For example, even without calling random functions, modern Python will have some randomness introduced, specifically into the hashing behavior for things that can be hashed as a series of bytes (bytes itself, str, datetime.datetime, etc.). While dicts have been insertion-ordered since 3.7 (3.6 as implementation-detail), sets are still arbitrarily ordered, and the ordering will change from run-to-run for any type using a seeded hashing scheme. If your code is assuming sets have a reproducible, meaningful order, and rely on that in some way, your runs could in fact change even if every other aspect of your code and inputs was deterministic.

Floating point math is order-sensitive as well; while purely deterministic code should do the floating point math in the same order, low-level high performance libraries might thread large batches of work, introducing non-determinism. And of course, if the sets mentioned earlier get involved, and their ordering determines the order of floating point math (float itself does not use a seeded hash, but a seeded hash type could influence float operation ordering depending on what you're doing), that will get some reliable non-determinism.

Of course, with code as huge as what you're doing, you might be using threading for a speed-up, and that would cause similar issues if you have any form of races.