experiment-designer

Topics in the source

• Experiment Designer
• When To Use
• Core Workflow
• Hypothesis Quality Checklist
• Common Experiment Pitfalls
• Statistical Interpretation Guardrails
• Tooling
• scripts/samplesizecalculator.py

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions. Use this skill for: A/B and multivariate experiment planning Hypothesis writing and success criteria definition Sample size and minimum detectable effect planning Experiment prioritization with ICE scoring Reading statistical output for product decisions 1. Write hypothesis in If/Then/Because format If we change [intervention] Then [metric] will change by [expected direction/magnitude] Because [behavioral mechanism] 2. Define metrics before running test Primary metric: single decision metric Guardrail metrics: quality/risk protection Secondary metrics: diagnostics only 3. Estimate sample size Baseline conversion or baseline mean Minimum detectable effect (MDE) Significance level (alpha) and power Use: 4. Prioritize experiments with ICE Impact: potential upside Confidence: evidence quality Ease: cost/speed/complexity ICE Score = (Impact Confidence Ease) / 10 5. Launch with stopping rules Decide fixed sample size or fixed duration in advance Avoid repeated peeking without proper method Monitor guardrails continuously 6. Interpret results Statistical significance is not business signif

This is an excerpt. Refer to the complete source and its current requirements.