A/B Testing Concepts and Challenges

The video tutorial explains the complexities of calculating sample size in A/B testing. It covers the derivation of the sample size formula, including the significance level, statistical power, and minimum detectable effect (MDE). The tutorial also clarifies the difference between lift and Delta squared, and how to estimate variance. It provides a practical example of calculating sample size for an A/B test, emphasizing the importance of understanding these concepts for product data science roles and interviews.

What is one of the main challenges in learning A/B testing?

• Interpreting the results

• Choosing the right software

• Calculating the sample size

• Understanding the user interface

What does the 'n' represent in the sample size formula?

• Number of variables

• Sample size per group

• Number of hypotheses

• Total sample size for the experiment

What is the significance level used for in hypothesis testing?

• To define the control group

• To set a threshold for statistical significance

• To determine the sample size

• To calculate the mean

What is statistical power?

• The probability of detecting an effect if it exists

• The probability of observing a p-value

• The probability of a Type I error

• The probability of a Type II error

How is Delta different from lift?

• Delta is a percentage improvement

• Delta is a relative change

• Delta is a statistical error

• Delta is an absolute difference

Why is estimating variance challenging before an experiment?

• Variance is constant across experiments

• Variance is not needed for sample size calculation

• Variance requires data from the treatment group

• Variance is always zero

In the practical example, what is the total sample size required for the experiment?

• 314

• 628

• 1000

• 200