I'm attempting to execute a C++ code on a cluster that utilizes Slurm. Since the server did not have g++ installed, I created a Singularity container (SIF) to facilitate the execution. However, I've observed considerable variations in the runtime of the same executable after each run. For instance, the code takes 1.30 hours to complete on the first run and then 2.30 hours on the second run. The cause of these fluctuations is unclear to me. Do you have any suggestions or insights on why this might be occurring?
To address this concern, I've attempted to modify the optimization flags, optimize the code, and profile its performance. However, despite these efforts, the issue persists.