analysis · Any · free
Compute the sample size I need for a legitimate A/B test on [METRIC]. Inputs: Baseline conversion rate or baseline mean + SD: [VALUE] Minimum detectable effect (relative or absolute): [%] Alpha (default 0.05): [VALUE] Power (default 0.80): [VALUE] Number of variants: [K] Weekly traffic to test surface: [N] Deliver: Required sample per variant (show the formula) Total sample across variants and weeks of traffic that translates to Trade off table: MDE at 1 week / 2 weeks / 4 weeks Whether my MDE is realistic vs vanity Multiple comparisons adjustment if K 2 (Bonferroni or better) What to do if I can't wait that long (sequential testing, priors, or don't test at all) Warn me if the test is underpowered before I run it.
#sample-size #power #ab-test