[ABTEST-500] Investigation: Data on result dashboard keeps changing for tests that reached conclusion Created: 24/Jun/21  Updated: 26/Aug/22

Status: Open
Project: A/B Testing
Component/s: None
Affects Version/s: 1.0
Fix Version/s: None

Type: Improvement Priority: Neutral
Reporter: Michael Duerig Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relation
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Documentation update required:
Yes
Date of First Response:
Epic Link: ABn Later
Story Points: 5
Team: AuthorX

 Description   

When a test concludes (either by finding significant results or by reaching its sample size), some of the data on the result tab still keeps changing when further visits/conversions happen. This affects

  • Unique visitors
  • Days running
  • Total conversions (also in the "Complete Test" dialog)
  • Conversion rate

At the same time the confidence rate refer to the data that was collected up to the point where the statistical test concluded and nothing thereafter. 

In the extreme case this can lead to the situation where initially variant A wins the test and is then subsequently overtaken by variant B. In this case the "Complete Test" dialog would determine variant B as winner, although statistically variant A is the winner.

 

Proposal: as soon as the statistical test concludes freeze the data on the result dashboard.

Note: update AB Test timeline if necessary.

 

 


Generated at Sun Feb 11 22:56:51 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.