Maximizing RoAI: Why Most Teams Are Measuring the Wrong Thing

Every leader I talk to these days is under pressure to "do something with AI." Boards ask about it. Investors expect a slide on it. Founders worry they're falling behind competitors that seem to ship AI features every week.

But here's what I keep running into: almost nobody can tell me what return they're actually getting. They can tell me how many AI features they've shipped. They can tell me how much they're spending on model API calls. What they can't tell me is whether any of it is making the product, the team, or the business meaningfully better.

I've started calling this RoAI, return on AI, and I think it's the metric most product organizations need to track. Not AI adoption. Not an AI feature count. Return. What did we get back for what we spent, in dollars, in time, in risk?

This matters in two very different places: the AI you're building into your product, and the AI you're using to build your product, including, increasingly, to run product management itself. The same RoAI question applies to both, but the risks are different, so let me take them one at a time and then connect them.

The Product Side: Features Are Not Outcomes

The first mistake I see constantly is treating "we added AI" as though it were the outcome itself. It's not. It's an input, same as any other engineering investment. The question was never "should we use AI?" The question should always be "what customer problem does this solve, and is AI genuinely the best way to solve it?"

I've seen teams bolt a chatbot onto a product because a competitor has one, with no clear job to be done, no success metric beyond "usage," and no plan for what happens when the novelty wears off. Six months later, engagement has cratered, the inference bill hasn't, and nobody wants to be the one to kill it.

The best product teams I know are relentless about working backward from the customer problem, the same discipline that's always separated strong product work from weak. They ask: is this a problem where probabilistic, generative output is actually valuable, or are we forcing a deterministic problem into a non-deterministic solution because it's trendy? A lot of "AI features" I review are really just poorly specified rules engines wearing an AI costume, and they'd be cheaper, faster, and more reliable as regular software.

When AI genuinely is the right tool: personalization, synthesis, natural-language interfaces, and pattern detection across noisy data, the teams getting real RoAI share a few habits:

• They instrument before they build. They define what "working" means: task success rate, time saved, error rate, and downstream retention impact. And they do so before a line of the feature ships, not after a QBR asks why usage is flat.

• They price in the ongoing cost, not just the build cost. An AI feature isn't done at launch. It has a variable cost of goods sold that scales with usage in a way traditional software mostly doesn't. A feature that delights users but destroys your gross margin isn't a win; it's a liability with good reviews.

• They treat model quality as a moving target, not a one-time decision. The model you built six months ago is not the model available today, and it won't be the model available in six months. Teams with real RoAI build evaluation harnesses and swap models when a better one arrives; teams without it are quietly running on outdated intelligence because nobody wants to touch the thing that "already works."

The Workflow Side: Automating Product Management Itself

The second front is a more sensitive one, because it's closer to home: using AI to automate the work of product management: writing specs, synthesizing research, drafting roadmaps, even making prioritization calls.

I want to be direct about this because I think it's where the most damage is being done right now, quietly, inside otherwise good companies.

AI is extraordinary at compressing the mechanical work of product management. Synthesizing forty customer interviews into themes. Drafting a first-pass PRD from a set of bullet points. Turning a messy roadmap doc into a clean stakeholder update. This is real, substantial time given back to product managers, and any team not using AI for this kind of compression is leaving obvious value on the table.

But there's a second category of work that looks similar on the surface and is completely different underneath: judgment. Deciding which of five plausible problems is actually the one worth solving this quarter. Weighing a customer's stated request against what you know about their actual underlying need. Making the call to kill a project in which the team has an emotional investment. This is the part of the job that AI cannot yet do well, because it requires context that doesn't live in any document. Things like conversations with your CEO about runway, a gut sense from years of watching this specific market, and an intuition about which stakeholder is optimizing for their own visibility rather than the customer.

The pitfall that I see constantly with founders trying to run lean is letting the first category quietly bleed into the second. A synthetic summary of user interviews feels like it's telling you what customers want. It's actually telling you what patterns the model found in the transcript, filtered through whatever confirmation bias was baked into the prompt. I've reviewed roadmaps where the prioritization "logic" was effectively laundering someone's pre-existing opinion through an AI tool to make it look objective. The output has the form of rigor without the substance of it.

The best teams I'm seeing draw a hard line here, and it's a useful one for any founder to steal: AI drafts, humans decide. Use it aggressively for synthesis, first drafts, and pattern-spotting. Never let it be the last checkpoint before a decision that involves trade-offs, trust, or resource allocation. The moment "the AI said so" becomes an acceptable answer to "why are we building this," you've outsourced the one part of the job you were actually hired to do.

The Pricing Problem Nobody's Planning For

Here's a topic that deserves more attention than it gets in most planning conversations: AI pricing is not stable, and treating it as a fixed line item is a planning error.

We've already watched per-token pricing drop dramatically over the past couple of years, which is great, except most teams' unit economics were built assuming today's prices, not tomorrow's. That cuts both ways. Prices for frontier capabilities can also spike when a new tier launches, when usage-based pricing replaces flat subscriptions, or when a vendor decides your use case is worth repricing once they see how you're actually using it. The teams getting burned right now are the ones that built a financial model on a snapshot instead of a trend line.

The best practice here is boring but effective: model your AI costs the way you would model any volatile input cost, with a range, not a point estimate. And revisit the model. Not annually, at least quarterly. If your product's margin structure only works at the current API pricing, you don't have a product; you have a bet.

The Dependency Risk Nobody Wants to Say Out Loud

The related and thornier issue is vendor dependency. Most companies building on AI today are building on someone else's model, roadmap, and business priorities. That's a reasonable trade when the alternative is spending years and tens of millions training your own foundation model. For the vast majority of product teams, it's the only sane choice. But "reasonable" doesn't mean "riskless," and I think a lot of leaders are underpricing that risk.

What happens to your product if your model provider deprecates the exact capability you depend on? If pricing changes make your unit economics unworkable overnight? If the vendor's own policies shift in a way that affects what your product is allowed to do?

Teams with real RoAI discipline don't try to eliminate this risk. They can't. But they actively manage it:

• They abstract the model layer. Prompts, evaluation harnesses, and business logic live separately from any single vendor's API, so switching providers is weeks of work, not a rewrite.

• They avoid single points of failure on capabilities core to the product. If a feature is genuinely core to the value proposition, they know what their fallback is if that specific capability disappeared or were to triple in price.

• They stay close to the roadmap, not just the API docs. The best product leaders I know treat their model vendor relationship the way they'd treat a critical infrastructure partner, with real visibility into where that partner is headed, not just what the current contract says.

Bringing It Together

The thread connecting all of this is one I've been saying for twenty years about product work generally, and it turns out to be exactly as true for AI: the tool doesn't produce the outcome; the judgment applied to the tool does. RoAI isn't a number you get by adding up API bills against feature launches. It's a discipline you instrument before you build, protecting the parts of the job that require real judgment, planning for cost and dependency volatility instead of being surprised by it.

The companies that will look back on this period as a genuine advantage won't be the ones that shipped the most AI features fastest. They'll be the ones who stayed disciplined about what problem they were solving, kept a human firmly at the point of every real decision, and treated their AI dependencies with the same rigor they'd apply to any other critical vendor relationship.