Evidence Designs

An evidence design is the way a reading's cause is established: by assigning who or where receives the activity and comparing, by modelling an outcome over time against the activity, or by dividing observed credit among the exposures that preceded an outcome. A reading names the design it is read by, and the attribution rules are one design among five.

What you can do with a evidence design

• tableEvidence Plan · Evidence
• tableEvidence Plan · Settings

Its evidence designs— 5

• Holdout experimentA randomised experiment in which some eligible people are withheld from the activity, and the difference in outcome between the exposed and the withheld is the effect.
• Geo experimentAn experiment in which whole geographies are assigned to receive or not receive the activity, and the difference in their outcomes is the effect.
• Matched-market testA comparison of markets that received the activity with markets chosen to resemble them that did not, without random assignment.
• Marketing mix modelA statistical model of an outcome over time against the activity across channels, from which each channel's contribution is estimated without assigning anyone.
• AttributionA rule that divides credit for an outcome across the exposures that preceded it, observed rather than assigned.

Why it matters

What goes wrong without it, as the commitments that license this kind say it, each standing on its sources.

• Cause is read by a declared designTwo readings of the same outcome, one from a randomised holdout and one from a last-touch rule, are not two estimates of one number; they are answers to different questions, and a dashboard that shows them side by side without saying which is which invites the wrong decision. The measurement literature is explicit that observed credit and assigned effect diverge, that the returns to advertising are small against the noise, and that the mix model is a third design with limits of its own. Naming the design on the reading, and typing the attribution rules into one design, is what lets a plan say what kind of evidence it is asking for.
  A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook · Causally motivated attribution for online advertising · Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing · Measuring Ad Effectiveness Using Geo Experiments · The Unfavorable Economics of Measuring the Returns to Advertising · Challenges and Opportunities in Media Mix Modeling