Stop Wasting Tokens

Most LLM cost is waste — context that didn’t need to be there, models too big for the task, reasoning that ran longer than it should. Here’s how to fix it, grounded in 2025 research, with a concrete open-source stack at the end.

The problem

Token cost and context bloat are the same problem: no mechanism deciding what information is worth keeping. It shows up three ways:

• Uncontrolled output. Thinking tokens are invisible but billed at 3–5x the input price. No budget cap = silent overbilling.
• Context bloat. Every API call sends the full conversation from scratch. History accumulates, quality degrades well before the limit.
• No routing. Simple classification tasks go to frontier models at frontier prices.

Fix these in order — output, then input structure, then context management, then routing. The savings compound.

Context degrades faster than you think

Models advertise 128K–1M token windows. Effective context — where accuracy stays near baseline — is 50–65% of that.

Don’t stuff documents in and hope. Treat context as a finite resource with non-linear quality decay.

Control what the model writes

Output tokens cost 3–5x more than input (sequential generation vs. parallel input processing). This is the highest-leverage place to start.

Cap max_tokens by task type. Classification: 16–32. JSON extraction: 128–256. Summarization: 256–512. Analysis: 512–1,024.

Control thinking budgets. DeepSeek-R1 averages ~12K thinking tokens per math question. Without a cap, reasoning pipelines cost 10–30x what you’d expect from visible output alone.

Use structured output. JSON schema reduces output ~15% and eliminates parsing retries.

Structure prompts for prefix caching

The model has no memory between calls. Every request sends the entire conversation as one flat token sequence — system prompt, all previous turns, new message. Every token gets KV-computed before the model can output anything.

Prefix caching reuses KV vectors when the first N tokens match a previous request. Anthropic gives 90% off cached tokens. OpenAI 50%. Google 75%. You just need to structure prompts so the static prefix is as large as possible.

The rule: static content first (system prompt, schema, few-shot examples), dynamic content last (conversation history, new message). Everything above the cache boundary must be byte-for-byte identical across requests.

# ❌ Cache-hostile — dynamic content at the top
messages = [
    {
        "role": "system",
        "content": f"""
            Current time: {datetime.now()}     # ← breaks cache every call
            User ID: {user_id}                 # ← breaks cache per user
            You are a helpful assistant...
            [800 tokens of static instructions]
        """
    },
    *conversation_history,
    {"role": "user", "content": user_message}
]

# ✓ Cache-friendly — static prefix locked, dynamic at bottom
messages = [
    {
        "role": "system",
        "content": """
            You are a helpful assistant.
            [Instructions, rules, schema — never changes]
            [Few-shot examples — never changes]
        """,
        "cache_control": {"type": "ephemeral"}  # Anthropic API
    },
    *conversation_history,
    {"role": "user", "content": user_message}
]

Skill templates take this further. For recurring tasks (reports, deployments, code reviews), the model spends thinking tokens rewriting the user’s request into a form it can reason about — every time. A pre-built template eliminates this translation cost. Combined with prefix caching, skill templates hit cache from the second call: 10% input cost, near-zero thinking overhead.

Manage context over time

Even with perfect prompt structure, context grows. Active management keeps quality high and costs linear.

Compaction. Summarize history into a compact block and reset. Proactive compaction beats automatic — the model’s recall is still intact and can be guided toward what matters. Write state to a JSON file, spawn a fresh context that reads it.

Subgoal-based chunking. HiAgent (ACL 2025) compresses completed subgoal history into short summaries. Result: 2x success rate on long-horizon tasks, 35% context reduction.

Hierarchical memory. Three layers: working memory (current window) → session memory (today’s summary) → long-term memory (persistent facts). The Letta/MemGPT framework validates this — the LLM self-manages context through function calls to page data in and out.

KG-RAG for history. Instead of keeping all history in context, use HippoRAG (NeurIPS 2024) to build a knowledge graph across sessions and retrieve by semantic query. 10–30x cheaper than iterative RAG.

Route intelligently

Routing isn’t just about sending easy queries to cheap models. It directly improves cache efficiency across the entire system.

Why routing multiplies savings

When a router dispatches to specialized sub-agents, each sub-agent gets a stable, task-specific prompt template and a fresh context window. Because every request to “Agent A” shares the exact same system prompt prefix, the KV cache is computed once and reused — cache hit rates approach 100% for the system prompt portion. The router effectively pre-warms cached templates for all downstream agents.

Two independent savings: cheaper models for simple tasks and better cache utilization for all tasks.

Tencent ADP: proof at scale

Tencent’s Agent Development Platform ships an Intent Classifier node (意图识别节点) as a first-class building block. User message enters → intent classifier (LLM with constrained prompt) produces a label → conditional branching routes to a specialized sub-graph with its own system prompt, tools, and knowledge base. Deployed across WeChat Pay and QQ Music customer service. The same pattern appears in ByteDance’s Coze, Baidu’s AppBuilder, and Dify — intent-based routing is now a standard production primitive.

The cache benefit is structural: each branch has a stable prompt template, so prefix caching kicks in automatically from the second request per intent category.

Building a router: four options

Start with: embedding similarity for high-confidence matches + local LLM for ambiguous cases + frontier API only when both are uncertain.

Open-source stack

User request
    │
    ▼
Qdrant semantic cache  ─── hit? → return cached response
    │ miss
    ▼
Ollama (Qwen2.5-1.5B)  ─── intent classification → skill_id
    │
    ▼
LiteLLM proxy
    ├── budget check (per-key caps)
    ├── inject skill template        ← prefix cached after first call
    ├── attach memory context        ← HippoRAG / Letta retrieval
    │
    ├── simple task  → SGLang (self-hosted 7B)
    ├── medium task  → RouteLLM decides
    └── complex task → frontier API (Anthropic / OpenAI)
              │
              ▼
        Letta memory manager
              ├── update session memory
              ├── extract to long-term store
              └── trigger compaction at 70% utilization

Start with LiteLLM + Ollama + Qdrant. That handles routing, semantic caching, and budget control. Add Letta and HippoRAG once you have multi-turn agents that need persistent memory.

What to do, in order

All combined: 90–95% savings vs. a naive single-model pipeline.

The most important thing: output tokens cost 3–5x more than input. Controlling what the model writes saves more than all input-side optimizations combined.

References

1. TALE — Han et al., ACL 2025, arXiv:2412.18547
2. Chain of Draft — Xu et al., arXiv:2502.18600, 2025
3. Unified Routing & Cascading — Dekoninck et al., ICLR + ICML 2025, arXiv:2410.10347
4. RouteLLM — Ong et al., ICLR 2025, arXiv:2406.18665
5. HiAgent — Hu et al., ACL 2025
6. NoLiMa — Adobe Research, ICML 2025, arXiv:2502.05167
7. LongBench v2 — Tsinghua, ACL 2025, arXiv:2412.15204
8. HippoRAG — Gutiérrez et al., NeurIPS 2024, arXiv:2405.14831
9. Context Rot — Chroma Research, 2025
10. Lost in the Middle — Liu et al., TACL 2024, arXiv:2307.03172
11. MemGPT / Letta — Packer et al., arXiv:2310.08560
12. SGLang — Zheng et al., NeurIPS 2024, arXiv:2312.07104
13. Effective Context Engineering for AI Agents — Anthropic Engineering Blog, 2025
14. AutoMix — Madaan et al., ICML 2024