Do you still need significance testing after A/B test, if the treatment effect is already larger than δ (effect size) used during power analysis?

Question

Assume we are designing an A/B testing. And we do the following steps:

1. Perform power analysis to set sample size using pre-defined δ (effect size), α and β.
2. Run A/B test using the sample size determined in step 1
3. Perform significant testing on A/B test result to see if treatment effect is significant.

The question is, if from step 2, we can see the treatment effect is already > δ, do we still need step 3?

Answer 1

You absolutely still need to do a significant test. This should be fairly trivial yet critical part of A/B test. In most cases, you should be able to borrow the work from the power calculation but I wouldn't shortcut and skip the significance test entirely. There are a few important reasons I can think of:

1. Standard error could be higher than expected, so even though your treatment effect is higher, it's possible that the results could still lack statistical significance.
2. Obtaining a p-value is an important result of an A/B test. Knowing how significant a result may not change your decision, but it could help in interpreting the experiment especially if you have to compare it against other A/B tests.
3. Understanding the confidence interval is important. All results are fuzzy, and knowing exactly how fuzzy or accurate your A/B test results is often useful for learning more about your treatment effect.

If you've already done the power analysis, running the statistical test should be fairly easy.