Details
-
Improvement
-
Resolution: Unresolved
-
Neutral
-
None
-
1.0
-
None
-
-
Empty show more show less
-
Yes
-
5
Description
When a test concludes (either by finding significant results or by reaching its sample size), some of the data on the result tab still keeps changing when further visits/conversions happen. This affects
- Unique visitors
- Days running
- Total conversions (also in the "Complete Test" dialog)
- Conversion rate
At the same time the confidence rate refer to the data that was collected up to the point where the statistical test concluded and nothing thereafter.
In the extreme case this can lead to the situation where initially variant A wins the test and is then subsequently overtaken by variant B. In this case the "Complete Test" dialog would determine variant B as winner, although statistically variant A is the winner.
Proposal: as soon as the statistical test concludes freeze the data on the result dashboard.
Note: update AB Test timeline if necessary.