Wattage Brings Cost-Regression Testing to AI Agent Development
Faizan Raza's open-source profiler analyzes OpenTelemetry traces to identify token waste and enforce spending thresholds in CI/CD pipelines.
AI agent development has a cost problem. Developer Faizan Raza built Wattage, an open-source profiler that treats token spend like any other engineering metric subject to regression testing.
Wattage analyzes OpenTelemetry (OTLP) JSON trace exports following OpenTelemetry GenAI semantic conventions. The tool operates fully offline and requires no API keys, suitable for teams concerned about sending trace data to external services.
Eight Detectors Target Specific Waste Patterns
Wattage identifies inefficiencies through eight detectors: prefix_churn, cache_gap, verbosity, redundant_tool_calls, nonconvergence, retrieval_thrash, model_mismatch, and reasoning_overspend.
The nonconvergence detector—dubbed the "convergence engine"—catches agent thrashing, including loops and oscillations that burn tokens without progress. Benchmarks in the repository show Wattage achieved an F1 score of 1.00 on this task, compared to 0.25 for SHA-256 exact-match detection.
Fixing a prefix_churn issue identified by Wattage reduced costs by 44.7%, dropping a single agent trace from $0.000199 to $0.000110. The savings compound across high-volume deployments.
CI Integration Enforces Cost Guardrails
Wattage's CI/CD integration can fail builds if an agent's cost increases beyond a defined threshold, functioning like performance benchmarking in traditional software. It supports GitHub Actions and emits SARIF reports for the GitHub Security tab, plus JUnit XML output for other CI systems.
The workflow uses a baseline.json file for cost regression comparison. Teams configure a separate workflow to update this baseline when changes merge to the default branch.
Visibility Through Flame Graphs and Badges
Wattage generates a shareable HTML flame graph of token spend via the --html flag, showing exactly where tokens are consumed. It also produces a README badge displaying the "Token Efficiency" score—a 0-100 grade for at-a-glance cost performance insight.
As AI agents move from prototypes to production, Wattage represents a shift from guessing about costs to engineering them. By quantifying waste in dollars and integrating cost checks into standard workflows, teams can optimize agent efficiency without manually auditing thousands of lines of traces.