Reader question: What would you ask an LLM when selecting software?
*** A quick note: We have a third webinar lined up, this time with Harry Rickerby from Briefly Bio. On January 16, 2025 at 2pm EST, we’ll be talking about Data you can trust: How Briefly helps biotech labs generate consistent, reproducible data. You can sign up here. ***
This week’s topic is a bit different from my recent posts, but if you bear with me, you’ll understand.
The tl;dr is that I’m on a mission to help software companies publish more detailed (and accurate) information about what their tools do, how they’re designed and how they’re used. But I need your help. In particular, I want to know:
If you had an LLM/AI chatbot with detailed, accurate and trustworthy information about software for biotech teams (or any other niche of your choosing), what questions would you ask it? What are the questions you’ve asked about software tools in the past, whether trolling around their websites, doing a formal committee-based evaluation/RFP or anything in between?
In the rest of this post, I’ll explain why I want to know this. If at any point you feel inspired to share, you can send me an email at jesse@merelogic.net, leave a comment on this post or ping me on Bits in Bio, LinkedIn or anywhere else we happen to be connected.
My main goal in creating the Biotech Reference Stack was to start collecting more detailed information about biotech software to help teams select the right tools. But as I’ve been writing more about LLMs, I’m realizing that there’s a potentially much bigger opportunity to shift how companies communicate this information.
In particular, it seems like folks are more and more going to use LLMs/AI chatbots to begin identifying and evaluating software. And sure, they’ll be able to go to the website or sign up for a demo to find any information that the LLM doesn’t know. But I believe that the more they can learn about a piece of software through the LLM, the more likely they will be to take that next step. (And the better experience it will be for everyone.)
In other words, I believe that it will become a competitive advantage for software companies to publish more detailed information that ChatGPT / Claude / Gemini / Perplexity / etc. can use to answer questions. I believe that getting detailed and reliable answers about a product will come to be an important sign of legitimacy, the equivalent of having a professional looking website or showing up at the top of the search results. And I believe the race to publish that information will ultimately be good for the folks (like you, dear reader) who want to make informed decisions without a huge hassle.
So I want to start evaluating how good these chatbots are today at answering the kinds of questions that potential users/buyers want to ask. To do that, I need a good understanding of what those questions are. And while I know what my own questions would be, I’m sure I’m missing a lot. So please, please, please - If you’ve read this far and you’ve got a couple of minutes, send me an email (jesse@merelogic.net) or a message through any means you like, to let me know what questions you would ask an LLM.