In this episode, Collin and I continue building our list of questions to help increase the chances of your AI investments delivering a return. We focus in particular on the role non-technical C-level leaders should play in this effort.

We use Chip Huyen’s AI Engineering as a practical framework, exploring how it gives digital leaders and non-technical senior managers a structured way to engage more deeply with AI initiatives. In particular, it provides a way into the ecosystem and lifecycle of AI application development—helping leaders ask better questions around things like data, prompt design, fine-tuning, and evaluation—so they can have more meaningful discussions with technical teams about how these applications are built, where the risks sit, and what criteria should be used to define and measure success.

We walk through some of the key differences between traditional software and AI systems—particularly the shift from deterministic to probabilistic behaviour, and the central role of data in shaping outcomes.

From there, we build on the questions we believe leaders should be asking: What problem are we solving? How are we evaluating outputs? How are we managing risks around data quality, safety, and factual accuracy? What trade-offs are we making between quality, cost, and latency?

We spend some time looking at Chip’s section on evaluation criteria, using it as a springboard for non-technical senior leaders to delve deeper into the thinking behind—and expected outcomes of—AI applications. We also introduce the concept of “evals”—ongoing evaluation frameworks that extend beyond traditional testing—and why they require continuous iteration, collaboration, and oversight, even after deployment.

This episode continues our exploration of how leaders can better understand what they are funding, engage more effectively with product and delivery teams, and create the conditions for AI investments to deliver real value.

Links to Chip's book & interview on evals referred to in the episode:

Chip Huyen’s AI Engineering: https://www.oreilly.com/library/view/ai-engineering/9781098166298/

Lenny’s podcast - Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar: https://www.youtube.com/watch?v=BsWxPI9UM4c

A significant number of AI initiatives are consuming budget and executive attention, while failing to reach production or deliver measurable value. Episode 12 of Talking Product helps you pressure-test your organisation's AI spend today.

In this episode, Collin and I use a Financial Times article as a springboard for a conversation about the state of AI initiatives in large organisations. The article—"AI's awfully exciting until companies want to use it"—captures a familiar pattern in technology: high expectations, significant investment, but limited impact. The article is freely available with FT registration.

In this episode, we explore three themes:

• Why pilots aren't scaling (often it's your data, not the technology)
• Whether organisations are bringing experimental rigour to AI adoption, or just buying impressive demos
• The leadership knowledge gap—understanding not just what AI can do, but what it can't

What you'll get:

• Questions that will help reveal whether your organisation is truly learning from its AI experiments or just spending
• Insight into why data governance problems you've been kicking down the road are now becoming existential
• What Gartner found about hidden costs in AI initiatives—and questions to ask to bring them out into the open
• Practical guidance on what experimental rigour actually looks like in an AI context

#AI #DigitalTransformation #Leadership #ProductManagement

In this episode, Collin shares a case about a leadership team building a hardware and software product.

We dig into this case because it highlights familiar product dynamics—great, passionate people, working to tight deadlines, but stuck in semi-siloed patterns and not really operating as one team. Each group pushes its own goals but isn’t fully aligned on the end game. And when the product leaders aren’t working effectively as a team, it has a real impact on outcomes and the ability to deliver at pace.

We chose this topic to give product leaders footholds and practical ideas for starting similar conversations in your own organisation. We often call ourselves a team—but are we, as product leaders, really operating as a team? What do we really mean when we use the word “team”?

In this session, Collin describes how he asked the leadership group that he was working with this simple, but powerful question: “Are you working as a team?” To their credit, they took the time to work with Collin and reflect on this question, as a collective.

One area of the case that I found interesting was when Collin unpacks the complexity of “owning” requirements as new information emerges or thinking shifts about a particular problem. It is this inherent complexity of digital product development that makes siloed, and even semi-siloed, ways of working so dangerous.

Collin also highlights a deeper systemic issue: senior and executive leaders often don’t grasp the importance of ways of working in digital product development. Instead, they simply push the silos to deliver faster. I personally feel that this level of naivety at the top is a serious competitive problem.

I hope you find this case useful. We welcome recommendations and comments—and we’re looking forward to getting a few more episodes out, sooner rather than later.

At the end of the podcast, I refer to the work of Nonaka & Takeuchi and said I would include a link: 1) The New New Product Development Game and 2) The Knowledge Creating Company are probably their most well known work. But there is a wealth of other material that is also worth exploring.