Model Selection: 3 methods you c/sh-ould use before an A/B test par Eustache Diemert

Doing randomized trial (e.g. A/B test) is the gold standard to prove that a new method or algorithm is better than the baseline. But such tests are costly and imply that the new method/implementation is of production quality. A natural solution is then to choose the better candidate offline using logged data.

We will explore different possibilities to do so and highlight their advantages and shortcomings. In particular, we will see that a family of methods for counter-factual reasoning are very close to what an A/B test could tell while operating on logged data.