🤔Limitations of Sequential Testing
Sequential hypothesis testing is a statistical analysis where the sample size is not fixed in advance, and the data are evaluated as they are collected. The analysis is terminated according to a predefined stopping rule as soon as significant results are observed. This allows you to draw conclusions at an earlier stage of the study than with more classic A / B testing, reducing the financial and time costs of the experiment.
To better understand the progress of sequential testing over the course of an experiment, it is helpful to think about thresholds that determine whether an effect is significant or not. These are commonly referred to as efficiency frontiers. When the Z-score calculated for the metric delta is above the upper bound, the effect is statistically positive. Conversely, a Z-score below the lower bound means a negative statistical signal result. At the beginning of the experiment, the efficiency margins are high. Intuitively, this means that a much higher significance threshold must be crossed in order to make an early decision when the sample size is still small. Borders are being adjusted every day. At the end of the predetermined duration, they reach the standard Z-score for the chosen significance level, for example: 1.96 for two-tailed tests with 95% confidence intervals.
While "peeping" into A/B testing is frowned upon, early monitoring of tests is critical to getting the most out of your experimentation program. If an experiment results in a measurable regression, don't wait until the end to take action. With sequential testing, statistical noise can be distinguished from strong effects that are significant at an early stage. Sequential testing also comes in handy when there are opportunity costs to run the experiment throughout its duration. For example, there is a significant engineering or business cost to failing an improvement from a subset of users, or when the completion of an experiment paves the way for further testing.
However, before making an early decision, it's worth remembering that even if one metric crossed the efficiency frontier, other metrics that appear neutral so far may be statistically significant at the end of the experiment. The efficacy frontier is useful for early determination of statistical results, but does not distinguish between no true effect and insufficient power before the target duration is reached.
The weekly seasonality of the experiments should also be taken into account. In particular, even when all the metrics of interest look great early on, it is recommended to wait at least 7 full days before making a decision. This is because many metrics are affected by weekly seasonality, with product end users behaving differently depending on the day of the week.
Finally, if a good estimate of the effect size is important, it is better to follow through with the experiment, as the adjusted confidence intervals of sequential testing are wider, so the range of likely values is larger when making an early decision. This means low accuracy. Also, a larger measured effect is more likely to be statistically significant early on, even if the true effect is actually smaller. Regularly making early decisions based on positive statistical results can lead to a systematic overestimation of the impact of running experiments, which also reduces accuracy.
https://blog.statsig.com/sequential-testing-on-statsig-a3b45dd8ab72