Is model interpretability a code smell?
OK, I’m finally getting back to posting about bio foundation models, informed by the other topics I’ve been writing about the last few weeks. In particular, I’ve been thinking about the processes and habits around data and AI in drug discovery. So when I saw this LinkedIn post by Nikhil Reddy Podduturi encouraging bio foundation model developers to create better interpretation tools, it got me wondering if this is a genuine technical need, or rather a symptom of a bigger issue related to the processes around these models. This week, I want to explain what I mean by this.
Model Interpretability
As AI/ML models get more complicated, it gets harder to tell how/why they came up with the predictions they did. With a linear model, you can look at which input features contributed the most to the final sum. With a multi-layer neural network, those input features get diced, folded, mixed and blended to the point where that’s no longer possible. For a transformer-based model where each token is generated by an algorithm involving multiple neural networks, it’s not even in the same universe as possible.
The idea of model interpretability is to introduce another algorithm that interprets the weights, outputs, etc. from a complex model to try and explain “why” it made the predictions that it did. There are a lot of different approaches to this, but they all introduce a fundamental risk: Whenever you try to reconstruct why something happened after the fact, it’s hard to tell if you’re learning something useful or just coming up with a good sounding story.
Code Smells
In software engineering, there’s this idea that you’ll sometimes see a pattern that isn’t necessarily bad on its own, but is rather a sign that something else is wrong. It’s awkward or unusual enough that if that’s the only way to solve a problem, there’s probably something bigger, and perhaps harder to identify, that’s the real problem. This is called a “code smell.”
I’m starting to think model interpretability is a code smell, at least in certain situations: It isn’t inherently bad, but it suggests there’s something missing from the information that you’re getting directly from the model. And if this is the case, the risk is that an interpretability layer will just trick you into thinking that it’s providing that information, without actually telling you what you need to know.
The Problem isn’t interpretability
So what if we stopped framing the problem as missing interpretability and instead posed it as a problem of trying to use these models for a purpose, or a process step, that they aren’t suited for.
We see a similar phenomenon with phenotypic screening, where drug candidates are initially identified by their impact on an in vitro assay, rather than directly observing which protein target(s) they bind to. Since the classical drug discovery process starts from identifying drug candidates that bind to a specific protein, it would be natural to want the phenotypic screening results to come with an interpretability tool that identifies the exact protein(s) causing the phenotypic result… but they don’t.
In fact, figuring out the protein target(s) that drugs from phenotypic screens bind to is incredibly difficult, complex and expensive. But that hasn’t stopped plenty of startups and pharma labs from using phenotypic screening. They just have to adjust their process to use the information that they do get from the phenotypic screens. And sometimes that means not knowing the binding protein(s).
Rethinking Processes
To be clear, I am not saying that we should trust the predictions/outputs of foundation models without any further scrutiny or validation. I mean, it should go without saying that any predictions will need to be verified in the lab many times over before any drug makes it to the clinic. But that validation is expensive and time consuming, so we also need a way to validate predictions digitally before we validate them in the lab.
What I’m saying instead is that we shouldn’t try to treat bio foundation models, or any other kind of model, as drop-in replacements for existing pieces of the drug discovery process and fill the gaps with (the illusion of) interpretability. Instead, we should evaluate the kind of information the models are providing and the level of confidence it provides, then rethink the processes around the model to accommodate that level of risk.
I don’t know exactly what that looks like, but I’m confident we can draw from examples like phenotypic screening to figure it out.
Thanks for reading! I help biopharma teams evolve their processes to extend the rigor they demand inside the lab to how they handle data and leverage AI outside the lab. If this is something you’re trying to figure out, visit merelogic.net to learn more.