Measure AI cost per successful, acceptable outcome, including retries and recovery. Token usage is a diagnostic input; it is not a substitute for user value or business impact.
Choose a denominator that matters
A cheap response that cannot be used is not an efficient outcome. Define what counts as success for the workflow: an answer that supports a decision, a completed task, or a change that passes the relevant checks.
Keep the quality threshold explicit. Otherwise, a team can make cost per task look better by accepting weaker results.
Count the whole path
Include failed attempts, retries, relevant infrastructure costs, and the effort required to inspect or repair results. Attribute shared costs consistently and explain which costs your measure omits.
For an illustrative internal assistant, compare workflows using the same task mix. Report the completion rate next to the cost measure. Do not compare one system on easy questions with another on difficult cases.
Optimize where the waste occurs
A high resource bill can reflect unnecessary context, repeated calls, weak instructions, or a poorly bounded task. Improving the model choice may help, but first inspect the journey.
Maintain a small evaluation set so changes in cost do not hide changes in quality. Segment by task type: the appropriate system for a brief summary may differ from the system needed for a consequential decision.
Bring the metric into prioritization
Use cost and usefulness together when deciding what to build next. An expensive workflow can be worthwhile when it reliably creates substantial value. A low-cost feature can still be wasteful when nobody needs it.
The product question is whether the outcome is worth the resources and risk it takes to achieve. Keep that question visible as implementation becomes faster.
Questions worth asking
Should teams optimize for the fewest tokens?
Optimize for useful outcomes within cost and quality constraints. Fewer tokens only help when they preserve the required result.
What costs are easy to miss?
Retries, evaluation, infrastructure, and human inspection or recovery can be omitted from a narrow inference-cost metric.
Sources & further reading
These sources inform the context. Recommendations and frameworks are editorial analysis, not claims of proven results. AI-assisted draft prepared for Yaron’s review; personal experience claims require his confirmation.