[ABTEST-500] Investigation: Data on result dashboard keeps changing for tests that reached conclusion Created: 24/Jun/21 Updated: 26/Aug/22 |
|
| Status: | Open |
| Project: | A/B Testing |
| Component/s: | None |
| Affects Version/s: | 1.0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Neutral |
| Reporter: | Michael Duerig | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Template: |
|
||||
| Acceptance criteria: |
Empty
|
||||
| Task DoD: |
[ ]*
Doc/release notes changes? Comment present?
[ ]*
Downstream builds green?
[ ]*
Solution information and context easily available?
[ ]*
Tests
[ ]*
FixVersion filled and not yet released
[ ] 
Architecture Decision Record (ADR)
|
||||
| Documentation update required: |
Yes
|
||||
| Date of First Response: | |||||
| Epic Link: | ABn Later | ||||
| Story Points: | 5 | ||||
| Team: | |||||
| Description |
|
When a test concludes (either by finding significant results or by reaching its sample size), some of the data on the result tab still keeps changing when further visits/conversions happen. This affects
At the same time the confidence rate refer to the data that was collected up to the point where the statistical test concluded and nothing thereafter. In the extreme case this can lead to the situation where initially variant A wins the test and is then subsequently overtaken by variant B. In this case the "Complete Test" dialog would determine variant B as winner, although statistically variant A is the winner.
Proposal: as soon as the statistical test concludes freeze the data on the result dashboard. Note: update AB Test timeline if necessary.
|