Theory vs Data

The cost of an experiment determines how much you should care about theory.

Before you run an experiment - whether a formal scientific one, or just trying something new - how confident should you be that you’ll learn something?

The best way to ensure an experiment will be successful is to leverage a theory about the experiment’s context.

A mental model. A rule of thumb.

From these you can form a hypothesis that the experiment will prove or disprove.

Theory makes experiments more reliable. But it also increases the cognitive load of experiment design. And if justifying the experiment with theory is formally required, it adds a time cost.

If your experiment is on the expensive side of the experiment cost inflection point, relying on theory is consistent with the tendency to push it towards reliability.

But if it’s on the inexpensive side, theory may just slow it down.

Why spend time building or applying a theory when you can directly measure the thing you care about?

If you can generate enough data with repeated experiments, theory becomes much less valuable.

How much are you leveraging or ignoring theory in the experiments you care about?