devtrev/actually-open-router
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(pricing): stop double-counting cached tokens; default cache prices to input rate #6

Merged
devtrev merged 1 commit from trevin/fix/cache-cost-double-counting into dev 2026-07-10 23:52:07 -06:00
Conversation 0 Commits 1 Files changed 13 +371 -127
devtrev commented 2026-07-10 23:20:42 -06:00
Owner
Copy link

Summary

  • Fix cache double-counting: cached input tokens were billed at both the full input rate AND the cache-read rate. OpenAI's prompt_tokens already includes cached_tokens (a subset), so the extractors now subtract cached from prompt so input holds only fresh billable tokens. Anthropic's counts are independent/additive (no subset), so no subtraction there.
  • Fix reasoning_tokens misnomer: completion_tokens_details.reasoning_tokens is a subset of output, NOT cache-creation input. It was wrongly mapped into cache_write (latent output-side double count for reasoning models). Now ignored.
  • Cache prices default to input rate: cache_read_per_million / cache_write_per_million changed from 0.0 to None; None resolves to input_per_million at billing time. Unconfigured cache tokens are never silently free. Explicit 0.0 still honored as "free cache".
  • New prompt_total_tokens column: nullable, migration-safe. Records total prompt for display parity with the OpenCode dashboard's Input column; input_tokens (fresh) shown as secondary.
  • Anthropic translator invariant: _convert_usage + streaming finalize() emit prompt_tokens = input + cache_read + cache_write (total) so the OpenAI extractor reads the correct total from translated bodies.

Root cause

OpenAI's usage.prompt_tokens is the total prompt tokens (fresh + cached). AOR stored it verbatim into TokenCounts.input and billed it at the full input rate, then also billed cached_tokens at the cache-read rate. Every cached token was overcharged by input_per_million. The fix is in the extract layer (per-adapter subset semantics), keeping compute_cost_cm a dumb sum.

Testing

  • uv run ruff check . — pass
  • uv run ruff format --check . — pass
  • uv run pytest -q — 291 passed

New tests in test_pricing.py:

  • test_cache_no_double_count_fixed_formula — known-input regression proving cached billed once at cache-read rate
  • test_cache_double_count_bug_would_overcharge — documents the pre-fix bug
  • test_cache_default_none_never_silently_free — cache prices default to input rate
  • test_translated_anthropic_cost_matches_native — invariant consistency across adapters
  • test_extract_usage_openai_reasoning_tokens_not_cache_write — reasoning no longer hijacked into cache_write
## Summary - **Fix cache double-counting**: cached input tokens were billed at both the full input rate AND the cache-read rate. OpenAI's `prompt_tokens` already includes `cached_tokens` (a subset), so the extractors now subtract cached from prompt so `input` holds only fresh billable tokens. Anthropic's counts are independent/additive (no subset), so no subtraction there. - **Fix reasoning_tokens misnomer**: `completion_tokens_details.reasoning_tokens` is a subset of output, NOT cache-creation input. It was wrongly mapped into `cache_write` (latent output-side double count for reasoning models). Now ignored. - **Cache prices default to input rate**: `cache_read_per_million` / `cache_write_per_million` changed from `0.0` to `None`; `None` resolves to `input_per_million` at billing time. Unconfigured cache tokens are never silently free. Explicit `0.0` still honored as "free cache". - **New `prompt_total_tokens` column**: nullable, migration-safe. Records total prompt for display parity with the OpenCode dashboard's Input column; `input_tokens` (fresh) shown as secondary. - **Anthropic translator invariant**: `_convert_usage` + streaming `finalize()` emit `prompt_tokens = input + cache_read + cache_write` (total) so the OpenAI extractor reads the correct total from translated bodies. ## Root cause OpenAI's `usage.prompt_tokens` is the **total** prompt tokens (fresh + cached). AOR stored it verbatim into `TokenCounts.input` and billed it at the full input rate, then *also* billed `cached_tokens` at the cache-read rate. Every cached token was overcharged by `input_per_million`. The fix is in the extract layer (per-adapter subset semantics), keeping `compute_cost_cm` a dumb sum. ## Testing - `uv run ruff check .` — pass - `uv run ruff format --check .` — pass - `uv run pytest -q` — 291 passed New tests in `test_pricing.py`: - `test_cache_no_double_count_fixed_formula` — known-input regression proving cached billed once at cache-read rate - `test_cache_double_count_bug_would_overcharge` — documents the pre-fix bug - `test_cache_default_none_never_silently_free` — cache prices default to input rate - `test_translated_anthropic_cost_matches_native` — invariant consistency across adapters - `test_extract_usage_openai_reasoning_tokens_not_cache_write` — reasoning no longer hijacked into cache_write 
Cached input tokens were billed at both the full input rate and the
cache-read rate because OpenAI's prompt_tokens already includes
cached_tokens (a subset), and compute_cost_cm charged both fields
independently. Every cached token was overcharged by input_per_million.

Root cause was in the extract layer, not compute_cost_cm: OpenAI/Google
report prompt_tokens as the TOTAL (fresh + cached), so the extractors
must subtract cached from prompt so 'input' holds only fresh billable
tokens. Anthropic's counts are independent/additive (no subset), so no
subtraction there — compute_cost_cm stays a dumb sum.

Secondary fix in the same pass: completion_tokens_details.reasoning_tokens
is a subset of completion_tokens (output), NOT cache-creation input. It
was wrongly mapped into cache_write, a latent output-side double count
for reasoning models. Now ignored (cache_write=0 for the OpenAI shape).

Cache pricing defaults changed from 0.0 to None; None resolves to
input_per_million at billing time. Unconfigured cache tokens are never
silently free (conservative fallback). Explicit 0.0 still honored as
'free cache'. This affects models with a pricing block but no explicit
cache_read/cache_write — they previously got cache tokens free.

New prompt_total_tokens column (nullable, migration-safe) records the
total prompt for display parity with the OpenCode dashboard's Input
column; input_tokens (fresh) shown as secondary. The Anthropic translator
now emits prompt_tokens = input + cache_read + cache_write (total) so
the OpenAI extractor reads the correct total from translated bodies.

Changes:
- pricing.py: TokenCounts.prompt_total; extract_usage_openai/google
  subtract cached from prompt; compute_cost_cm resolves None cache
  prices to input_per_million; extract_usage_anthropic sets prompt_total
- config.py: cache_read/cache_write_per_million default to None
- translate/anthropic.py: _convert_usage + streaming finalize() emit
  prompt_tokens as total; drop reasoning_tokens->cache_write mapping
- proxy.py: _drain_sse_usage + _maybe_extract_stream_usage mirror the
  buffered change; pass prompt_total_tokens through logging
- usage.py, db.py: prompt_total_tokens column + migration + wiring
- stats.py: SUM(prompt_total_tokens) in key summary + per-model + recent
- templates/key_detail.html: in/out column shows prompt_total/out,
  fresh input as secondary metric
- tests: regression tests for the double-count, cache-default-to-input,
  translated-vs-native cost equivalence; updated proxy/translate tests
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
devtrev/actually-open-router!6
No description provided.