Planning, Running, and Analyzing Controlled Experiments on the Web

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, and MultiVariable Tests (MVT). Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. Data Mining and Knowledge Discovery techniques can then be used to analyze the data from such experiments. The tutorial will provide a survey and practical guide to running controlled experiments based on the recently published survey article in the Data Mining and Knowledge Discovery Journal, co-authored with the two of the tutorial co-presenters [[http://exp-platform.com/dmkd_survey.aspx|Controlled Experiments on the Web: Survey and Practical Guide]], and based on the book “Always Be Testing” co-authored by the 3rd tutorial co-presenter [[http://www.amazon.com/Always-Be-Testing-Complete-Optimizer/dp/0470290633|Always Be Testing: The Complete Guide to Google Website Optimizer]]. The book includes use of industry tools, such as Google Website Optimizer and recently ranked #2 on Amazon’s sales rank for computers/e-commerce books. The tutorial includes multiple real-world examples of actual controlled experiments (many with surprising results), a review the theory and the statistics used to plan and analyze such experiments, and a discussion of the limitations and pitfalls that might face experimenters. Demos will be shown of some tools that support controlled experiments.