One of the consistent promises of AI-driven drug development is that it will cut development time of a new drug from over a decade down to a handful of years. But every new methodology requires iteration, particularly those involving ML. So how do you iterate on a years- or decades-long process?
When it comes to a new drug, the only test that really matters is the phase-three clinical trial, which can take years and cost millions of dollars. So most of the iteration in classical drug development uses faster and cheaper tests that help predict if the clinical trial will succeed. These early tests are proxies, and typically the more accurately a proxy predicts, the more expensive it will be in terms of both time and money.
So drug development follows a succession of increasingly expensive proxies that knock more and more drug candidates out of the pipeline. In-vitro tests in cells that replicate disease conditions and measure results increasingly accurately. In-vivo tests in animal models that are increasingly similar to humans. Each proxy raises the overall cost per candidate, but reduces the number of more expensive failures for the next proxy.
These proxies seem like great news for ML models that rely on cheap training data. There's nowhere near enough clinical trials, but plenty of proxy data. However, there's a catch: It's possible that these models are "overfitting" to the proxies. It's possible they're learning how to find drug candidates that will fool the early, cheap proxies, but won’t actually cure the disease. Instead, they’ll make it farther down the pipeline, maybe even to clinical trials, before getting knocked out.
Luckily, this isn't a new problem. This is exactly why drug development has evolved into this series of proxies. But for the proxies to do their job, they need to be tuned to the types of bias that you expect from the candidate selection approach. And you can’t do that unless the wet lab and data scientists are on the same page. In other words, they need a shared mental model of what proxies are available, what types of bias they need to account for, and how they can be used to prevent the ML models from cheating.
Without shared mental models, you may be able to speed up time to clinical trial, but at the risk of even more failed trials.