post date: 2020-3-2
Also known as bucket testing or split-run testing, is a user experience research methodology.
A/B test is a randomized experiment with two variants, A and B. Are considered the simplest form of controlled experiment.
A and B have difference in a single variable, that might affect a user's behavior. The benefits of A/B testing are considered to be that it can be performed continuously on almost anything.
example
- Landing-page optimization
- Ad creative optimization
statistic tests
- z-test
- z = (p_A-p_B)/(\sqrt{SE_A^2+SE_B^2})
- SE_A^2=p_A(1-p_A)/n_A
- SE_B^2=p_B(1-p_B)/n_B
- normal distribution
- t-test (equal but unkonwn variances)
- t = ((X1-X2)-(\mu1-\mu2))/(\sqrt{(s_p^2/n1+s_p^2/n2)})
- df = n1+n2-2
- s_p^2 is a weighted average of the sample variance
- s_P^2=((n1-1)s1^2+(n2-1)s2^2)/(n1+n2-2)
- t distribution
- t-test (unequal and unkonwn variances)
- t = ((X1-X2)-(\mu1-\mu2))/(\sqrt{(s_1^2/n1+s_2^2/n2)})
- df = (s1^2/n1+s2^2/n2)^2/(s1^4/n1^3+s2^4/n2^3)
- s1, s2 is the sample variance
- t distribution