Uploaded image for project: 'A/B Testing'
  1. A/B Testing
  2. ABTEST-500

Investigation: Data on result dashboard keeps changing for tests that reached conclusion

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Neutral
    • None
    • 1.0
    • None
    • Yes
    • 5

    Description

      When a test concludes (either by finding significant results or by reaching its sample size), some of the data on the result tab still keeps changing when further visits/conversions happen. This affects

      • Unique visitors
      • Days running
      • Total conversions (also in the "Complete Test" dialog)
      • Conversion rate

      At the same time the confidence rate refer to the data that was collected up to the point where the statistical test concluded and nothing thereafter. 

      In the extreme case this can lead to the situation where initially variant A wins the test and is then subsequently overtaken by variant B. In this case the "Complete Test" dialog would determine variant B as winner, although statistically variant A is the winner.

       

      Proposal: as soon as the statistical test concludes freeze the data on the result dashboard.

      Note: update AB Test timeline if necessary.

       

       

      Checklists

        Acceptance criteria

        Attachments

          Activity

            People

              Unassigned Unassigned
              mduerig Michael Duerig
              AuthorX
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Checklists

                  Task DoD