Are your AI agents wasting money? Understanding and controlling token costs

The fundamental unit of cost for large language models is the token. While simple AI queries have predictable costs, the multi-step, self-correcting nature of agentic workflows can cause these costs to spiral unpredictably. In a recent discussion, the AI Daily Brief highlighted the problem of "tokens that spin"—wasteful computation where an agent fails to complete a task effectively. For businesses, the key is to shift focus from the raw cost of tokens to the cost per successful task, enabling better cost control and a clearer view of AI-driven value.

The AI Daily Brief recently provided a practical guide for operators on understanding and managing the costs associated with large language models (LLMs), focusing on the concept of tokens.

What are tokens?

A token is the basic unit of data that an LLM processes. For English text, a token is roughly equivalent to four characters or about three-quarters of a word. Every interaction with an LLM, from the prompt you provide (input) to the model's response (output), is measured and billed in tokens. Different models have different pricing per million tokens for input and output.

Why agentic workflows multiply costs

While the cost of a single prompt-and-response cycle is often trivial, agentic AI workflows introduce a new level of complexity and cost. An AI agent designed to complete a task like "research and summarise the latest financial reports for ASX-listed mining companies" does not perform a single action. It undertakes a sequence of steps:

1. Deconstructs the request: What am I being asked to do?
2. Forms a plan: I will search for ASX mining companies, then search for their latest reports, then read each report, then extract key figures, and finally synthesise a summary.
3. Executes the plan: It uses tools (like web search), reads results, and generates internal thoughts or 'monologues' at each stage.
4. Self-corrects: If a search fails or a document is irrelevant, the agent re-evaluates and tries a new approach.

Each of these steps consumes tokens. When an agent gets stuck in a loop, repeatedly tries a failing strategy, or generates excessive internal monologue without making progress, it results in what the briefing called “tokens that spin”—computational effort that generates cost but no value.

This highlights how agentic systems, while powerful, can quickly lead to unpredictable and escalating operational expenses if not managed carefully.

For any business owner or operator deploying AI, moving from simple chatbots to sophisticated agents changes the cost management game entirely. Ignoring the dynamics of token consumption in agentic systems is a direct threat to your project's return on investment.

The metric that matters: Cost per successful task

The most critical takeaway is the need to shift focus from cost per token to cost per successful task. A cheap workflow that fails 50% of the time is more expensive than a pricier one that succeeds consistently. Your key performance indicator should be the total token cost required to achieve a desired business outcome. This reframes the objective from minimising raw token usage to optimising for valuable, completed work.

The risk of unmonitored agents

Without proper oversight, AI agents can become a financial black hole. An agent stuck in a loop can burn through a significant budget in hours, producing nothing of value. This isn't a theoretical risk; it's a practical operational challenge that requires new monitoring and observability practices. You cannot manage what you cannot measure, and traditional application performance monitoring tools are not equipped to analyse the internal state and token consumption of an AI agent.

Model selection becomes strategic

This cost-consciousness forces a more strategic approach to model selection. Using the most powerful—and most expensive—model like GPT-4o or Claude 3 Opus for every single step is inefficient. A more sophisticated and cost-effective strategy is to use a model cascade:

• Router/Classifier: Use a small, fast, and cheap model (e.g., Llama 3 8B, Claude 3 Haiku) to perform an initial analysis of a task.
• Escalation: If the task is simple (e.g., reformatting text), the cheap model handles it. If it's complex (e.g., multi-step reasoning and research), it is escalated to a more powerful model.

This tiered approach ensures you are only paying for high-end reasoning capabilities when they are genuinely required, dramatically reducing the overall cost per task.

Controlling agentic AI costs doesn't mean halting experimentation. It means implementing a framework for visibility and control. Here are the practical steps businesses can take today.

1. Audit Your AI Usage: Begin by identifying all workflows in your organisation that use LLMs, particularly those with agentic or multi-step characteristics.

2. Define 'Success' and 'Failure': For each workflow, establish clear, measurable criteria for what constitutes a successful outcome. This is the denominator in your most important metric.

3. Implement Token Logging and Monitoring: You cannot manage costs without data. Ensure your systems are logging the token consumption (both input and output) for every step of your agentic processes. Use this data to calculate your cost per successful task.

4. Set Up Anomaly Detection and Budget Alerts: Configure your monitoring tools to alert you to unusual spikes in token usage. This is your primary defence against "spinning tokens" and agents caught in expensive loops. Set hard budget limits for specific workflows or projects to prevent catastrophic overruns.

5. Adopt a Model Cascade Strategy: Review your agentic workflows and identify steps that can be handled by smaller, more economical models. Implement a routing system that allocates tasks to the most appropriate model based on complexity, reserving your most powerful models for the most demanding work.

6. Create a Budgeted Sandbox for Innovation: Don't let cost concerns kill innovation. Allocate a separate, ring-fenced budget for your R&D and product teams to experiment with new agentic systems. This allows for creative exploration within a controlled financial environment, ensuring that learning from failures doesn't jeopardise your operational budget.