Metrics and Proxies

Reducing experiment costs by changing what you measure

In my last blog post, I explored how the success of a biotech research platform depends on managing the cost - in terms of time, mental energy and money - of the repeated experiments that make up the Experiment Factory.

One way to do this is to introduce smaller, pre-experiments that make the expensive experiment more reliable (but overall more expensive).

They do this by measuring something different from, but closely correlated with, the thing that the expensive experiment measures.

This is called a proxy metric.

In drug development, the ultimate experiment is the Phase 3 clinical trial that determines if the drug can go to market.

A Phase 2 clinical trial measures the effect of a drug on a small group of people, which is strongly correlated with its effect on the larger group in the Phase 3 trial.

An animal study measures its effect on a different species altogether, which is a less effective proxy than a Phase 2 trial, but more effective than a less expensive in-vitro experiment.

In fact, the larger effect that the Phase 3 clinical trial measures is a proxy for the drug’s effect on the whole patient population - an even more expensive experiment that we really REALLY care about.

You use a proxy metric when measuring the thing you really care about is too expensive.

The proxy metric can tell you how to better frame the larger experiment, or even tell you not to run it at all.

But to effectively define use proxies, you need to recognize them as proxies, understand the nature of the correlation, and plan accordingly.