I am considering profiling a Python application in production (such as a Django website). I've found many options that self-describe as lightweight and demonstrate how they are used (including cProfile, vmprof, yappi and DTrace/SystemTap) but I am struggling to get a sense of what kind of negative performance impact I can expect while the profiler is in operation. This would be an important factor in identifying whether a particular tool is suitable for profiling in production.
Is it possible to anticipate the performance impact of a particular profiler, without experimentation in a test environment?
A profiler will not in itself speed up you program what it does is tell you where the program spends most time. Suppose that you profiler tells you that the program is in func1 1% of the time and in func2 80% of the time. If you make func1 run twice as fast you will speed up your progrram by about .5%. If you make func2 run twice as fast you will speed up you program by 40%. Spend your time and effort on func2.
How do you make functions faster? There is no simple answer, but there are lots of sites that talk about this (search for Python Performance). What sort of performance increase can you expect? Again there is no simple answer. My rule of thumb is a factor of two increase, but your mileage will probably vary?