cost-aware-llm-pipeline

Topics in the source

• Cost-Aware LLM Pipeline
• When to Activate
• Core Concepts
• 1. Model Routing by Task Complexity
• 2. Immutable Cost Tracking
• 3. Narrow Retry Logic
• 4. Prompt Caching
• Composition

Patterns for controlling LLM API costs while maintaining quality. Combines model routing, budget tracking, retry logic, and prompt caching into a composable pipeline. Building applications that call LLM APIs (Claude, GPT, etc.) Processing batches of items with varying complexity Need to stay within a budget for API spend Optimizing cost without sacrificing quality on complex tasks Automatically select cheaper models for simple tasks, reserving expensive models for complex ones. Track cumulative spend with frozen dataclasses. Each API call returns a new tracker — never mutates state. Retry only on transient errors. Fail fast on authentication or bad request errors. Cache long system prompts to avoid resending them on every request. Combine all four techniques in a single pipeline function: | Model | Input ($/1M tokens) | Output ($/1M tokens) | Relative Cost | |-------|---------------------|----------------------|---------------| | Haiku 3.5 (legacy) | $0.80 | $4.00 | 0.8x | | Haiku 4.5 | $1.00 | $5.00 | 1x | | Sonnet 5 | $2.00 | $10.00 | 2x | | Sonnet 4.6 | $3.00 | $15.00 | 3x | | Opus 4.8 | $5.00 | $25.00 | 5x | | Fable 5 / Mythos 5 | $10.00 | $50.00 | 10x | | Opus 4.0 / 4.1 (legac

This is an excerpt. Refer to the complete source and its current requirements.