AI Product Manager

A product manager ships an AI feature the way they have shipped every feature before: define the requirement, agree acceptance criteria, verify the build matches the spec, launch. Three weeks later the same input produces a different output, the per-user cost is four times the model in the business case, and quality has quietly drifted since the vendor updated the model. Nothing here was a mistake in execution. The category of thing being managed changed, and the process did not.

Four things that genuinely differ

Outputs are probabilistic. Traditional software either meets its acceptance criteria or does not. An AI feature has a quality distribution, and the interesting decisions live in the tail. "It works" stops being a meaningful statement; "it produces an acceptable answer 94 percent of the time, and here is what the other 6 percent look like" is the replacement.

Cost is per interaction. Classical software costs are dominated by building it once. An AI feature charges you every time a user touches it, and heavy users cost dramatically more than light ones. This drags unit economics forward from a finance concern into a design constraint — a feature can be excellent and still be unshippable at its price.

Quality drifts without a deploy. A model version changes, the input distribution shifts as users learn what to type, or upstream data moves. Your code did not change and behaviour did. There is no equivalent to this in classical PM.

Evaluation is a product artifact. The eval set is not a QA deliverable, it is the operational definition of what "good" means for this feature, and writing it is a product decision about which failures are acceptable and which are not.

Why the cycle looks different

The most practically important consequence: feasibility is discovered, not estimated. In classical PM, given a clear spec, an engineer can estimate. With AI features, whether the model can do the task at acceptable quality is often unknown until someone tries it on real data.

This inverts the usual sequence. A quick prototype against real inputs comes before detailed design, because the design depends on a capability you cannot assume. Teams that write a full PRD first routinely find the central assumption was wrong and the document was fiction from the first page.

Failure modes

Demo-driven confidence — a compelling demo on curated inputs sets expectations the production distribution cannot meet. Signal: launch quality dramatically below what stakeholders saw.

Unmodelled unit economics — pricing set before measuring cost per interaction. Signal: gross margin falling as adoption rises, so success makes things worse.

Silent drift — quality degrades with no deploy and no alert. Signal: complaints rising while dashboards show a healthy system.

Single-number quality — one accuracy figure hides the failure tail. Signal: a good headline metric alongside a specific, repeated user complaint nobody can locate in the data.

Explaining this to stakeholders

On probabilistic output: classical software is a calculator, which is right every time or broken. An AI feature is a very capable new colleague — usually excellent, occasionally confidently wrong, and needing a way to check.

On per-interaction cost: this feature has something closer to a marginal cost of goods, like shipping a physical item. Every use costs money, so the more successful it is, the larger the bill.

On why estimates carry more uncertainty: we are not estimating how long it takes to build a known thing. We are finding out whether the thing is possible at the quality we need, and that answer arrives by trying.