devtrev/actually-open-router
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: cache-warmth routing (sticky, backoff, thresholds, retry budget) #1

Merged
devtrev merged 7 commits from trevin/feat/sticky-backoff-routing into dev 2026-07-09 20:17:05 -06:00
Conversation 0 Commits 7 Files changed 18 +1866 -151
devtrev commented 2026-07-09 19:46:51 -06:00
Owner
Copy link

Summary

Implements the cache-warmth routing redesign specified in docs/requirements-sticky-backoff.md. The v1 router always tried providers in a fixed priority order with no memory of recent failures, so a persistently rate-limited primary provider was retried first on every request — burning the prompt cache on the healthy fallback and never warming the one it kept hitting.

This adds sticky routing (per-client+model last-success target tried first), backoff (per-provider+key exponential suppression after 429/5xx/network errors), two cache-threshold knobs for re-probing preferred providers, and a max_opaque_retries budget for intra-request walking.

What changed

Config decoupling (commit 1) — API keys are now declared in a top-level [api_keys.<name>] section (with env, priority, optional name) and referenced from providers via keys = [...]. This gives keys a stable cross-restart identity used by sticky/backoff. Legacy inline [[providers.*.api_keys]] is rejected at startup with an actionable error pointing at the spec. Validation rules 9-12 added.

DB schema (commit 2) — Two new sqlite tables: sticky(client_key_hash, canonical_model, provider, key_name, last_success_ts, PK(client,model)) and backoff(provider, key_name, until_ts, last_failure_ts, window_s, failure_count, last_error_kind, PK(provider,key)). Both created idempotently.

Backoff module (commit 3) — aor/backoff.py: record_failure, is_backed_off, get, clear, clear_provider, clear_all, list_active. Exponential doubling curve (5s initial, x2 per failure, 24h cap). Success drops the record and resets the counter. Lazy expiry (rows past until_ts treated as absent). Shared across all clients (one record per provider+key).

Sticky module (commit 4) — aor/sticky.py: get, set_on_success, list_all. Per-(client,model) last-success target, persisted to sqlite so it survives restarts. Sticky only changes to a different (provider,key) when that candidate succeeds — never merely because a preferred candidate failed.

Routing rewrite (commit 5) — aor/routing.py select_sequence now: (1) computes non-sticky ordering, (2) filters out backed-off candidates, (3) reads sticky, (4) if sticky is live and its (provider,key) still in config → tries it first, (5) applies threshold logic to decide whether to re-probe a preferred candidate vs stick with the warm cache. Two new [routing] knobs: provider_priority_cache_threshold and key_priority_cache_threshold (both default 0 = always re-probe a higher-priority candidate when it's available).

Proxy integration (commit 6) — aor/proxy.py: on success records sticky + clears backoff for that (provider,key); on retryable failure records backoff if the error is 429/5xx/network (402/quota-403 are retryable for walking but do NOT trigger backoff). New max_opaque_retries knob (default unlimited = v1 walk-whole-list behavior; 0 = first-attempt-only). When all candidates are backed off → terminal HTTP 429 with aor_all_backed_off error code. Sticky/backoff recording is independent of auto_fallback.

Admin UI (commit 7) — Backoff panel on the stats detail view (per-provider) + active-backoff summary on the overview. Reset-backoff (per-provider) and clear-all buttons, both session-gated, with a flash confirmation message.

Testing

  • uv run ruff check . — clean
  • uv run ruff format --check . — clean (30 files)
  • uv run pytest -q — 135 passed

New test files: tests/test_db.py, tests/test_backoff.py, tests/test_sticky.py. Existing test config strings rewritten to the new [api_keys.*] shape. New proxy tests cover sticky persistence, backoff triggers (429/5xx/network vs 402), terminal aor_all_backed_off, max_opaque_retries budget (0 / limited / unlimited), and auto_fallback=false independence.

Spec reference

Full design and decision rationale: docs/requirements-sticky-backoff.md (merged to dev separately in c050d32).

## Summary Implements the cache-warmth routing redesign specified in `docs/requirements-sticky-backoff.md`. The v1 router always tried providers in a fixed priority order with no memory of recent failures, so a persistently rate-limited primary provider was retried first on every request — burning the prompt cache on the healthy fallback and never warming the one it kept hitting. This adds sticky routing (per-client+model last-success target tried first), backoff (per-provider+key exponential suppression after 429/5xx/network errors), two cache-threshold knobs for re-probing preferred providers, and a `max_opaque_retries` budget for intra-request walking. ## What changed **Config decoupling (commit 1)** — API keys are now declared in a top-level `[api_keys.<name>]` section (with `env`, `priority`, optional `name`) and referenced from providers via `keys = [...]`. This gives keys a stable cross-restart identity used by sticky/backoff. Legacy inline `[[providers.*.api_keys]]` is rejected at startup with an actionable error pointing at the spec. Validation rules 9-12 added. **DB schema (commit 2)** — Two new sqlite tables: `sticky(client_key_hash, canonical_model, provider, key_name, last_success_ts, PK(client,model))` and `backoff(provider, key_name, until_ts, last_failure_ts, window_s, failure_count, last_error_kind, PK(provider,key))`. Both created idempotently. **Backoff module (commit 3)** — `aor/backoff.py`: `record_failure`, `is_backed_off`, `get`, `clear`, `clear_provider`, `clear_all`, `list_active`. Exponential doubling curve (5s initial, x2 per failure, 24h cap). Success drops the record and resets the counter. Lazy expiry (rows past `until_ts` treated as absent). Shared across all clients (one record per provider+key). **Sticky module (commit 4)** — `aor/sticky.py`: `get`, `set_on_success`, `list_all`. Per-(client,model) last-success target, persisted to sqlite so it survives restarts. Sticky only changes to a different (provider,key) when that candidate succeeds — never merely because a preferred candidate failed. **Routing rewrite (commit 5)** — `aor/routing.py` `select_sequence` now: (1) computes non-sticky ordering, (2) filters out backed-off candidates, (3) reads sticky, (4) if sticky is live and its (provider,key) still in config → tries it first, (5) applies threshold logic to decide whether to re-probe a preferred candidate vs stick with the warm cache. Two new `[routing]` knobs: `provider_priority_cache_threshold` and `key_priority_cache_threshold` (both default 0 = always re-probe a higher-priority candidate when it's available). **Proxy integration (commit 6)** — `aor/proxy.py`: on success records sticky + clears backoff for that (provider,key); on retryable failure records backoff if the error is 429/5xx/network (402/quota-403 are retryable for walking but do NOT trigger backoff). New `max_opaque_retries` knob (default unlimited = v1 walk-whole-list behavior; `0` = first-attempt-only). When all candidates are backed off → terminal HTTP 429 with `aor_all_backed_off` error code. Sticky/backoff recording is independent of `auto_fallback`. **Admin UI (commit 7)** — Backoff panel on the stats detail view (per-provider) + active-backoff summary on the overview. Reset-backoff (per-provider) and clear-all buttons, both session-gated, with a flash confirmation message. ## Testing - `uv run ruff check .` — clean - `uv run ruff format --check .` — clean (30 files) - `uv run pytest -q` — 135 passed New test files: `tests/test_db.py`, `tests/test_backoff.py`, `tests/test_sticky.py`. Existing test config strings rewritten to the new `[api_keys.*]` shape. New proxy tests cover sticky persistence, backoff triggers (429/5xx/network vs 402), terminal `aor_all_backed_off`, `max_opaque_retries` budget (0 / limited / unlimited), and `auto_fallback=false` independence. ## Spec reference Full design and decision rationale: `docs/requirements-sticky-backoff.md` (merged to `dev` separately in c050d32). 
Move key definitions out of inline [[providers.*.api_keys]] arrays into a
new top-level [api_keys.<name>] table where each key declares its env var,
priority, and optional friendly display name. Providers now reference keys
by name via a required `keys = [...]` list.

ResolvedKey is reshaped to (key_name, env, display_name, value, priority):
key_name is the stable cross-restart identity for the upcoming
sticky/backoff work; display_name is purely cosmetic for the admin UI.

Startup validation enforces (spec §8 rules 9-12):
- each [api_keys.*] env var is present and non-empty
- no two [api_keys.*] entries reference the same env var (rule 12)
- every provider.keys reference resolves to an existing api_keys name (rule 10)
- every provider declares at least one key and one model

The legacy inline [[providers.*.api_keys]] format is now hard-rejected at
startup with an actionable error citing docs/requirements-sticky-backoff.md
§5, run before pydantic parse so the operator gets the spec link rather
than an opaque pydantic message.

config.example.toml and all test config strings rewritten to the new shape.
routing.py keeps the v1 consistent-hash algorithm; only the within-provider
hash tuple changes from str(k.index) to k.key_name. The full sticky+backoff
routing rewrite is a later commit.
Add two new sqlite tables to the db schema:
- sticky(client_key_hash, canonical_model, provider, key_name,
  last_success_ts, PK(client,model)) — per-(client,model) last-success
  target so repeated requests from one client hit the same upstream prompt
  cache.
- backoff(provider, key_name, until_ts, last_failure_ts, window_s,
  failure_count, last_error_kind, PK(provider,key_name)) — per-(provider,
  key) shared backoff state; expiry is lazy (until_ts<now treated as
  absent, not proactively deleted).

The sticky composite PK already auto-indexes its only access path, so no
separate index is created there. backoff gets idx_backoff_until for the
admin "currently backed off" view (spec §7); the spec disclaims any
proactive expiry scan.

tests/test_db.py covers table/index presence, idempotent re-init, and PK
uniqueness enforcement (one row per pair) via IntegrityError assertions.
aor/backoff.py manages shared backoff state persisted to the backoff
sqlite table. One row per (provider, key_name), global across all
clients (rate limits are per subscription, not per client).

Curves per spec §3: first failure backs off 5s; each subsequent failure
while still backed off doubles the window, hard-capped at 24h; a failure
arriving after the previous window elapsed naturally resets to 5s. The
reset predicate uses `until_ts <= now` so it agrees with is_backed_off's
strict `until_ts > now` active predicate at the exact expiry instant —
a probe the router has already unlocked starts a fresh 5s window on
failure rather than escalating an expired one.

Expiry is lazy: rows with until_ts <= now are filtered out at read time
and never proactively deleted. On success clear() drops the row and the
counter resets (next failure starts fresh).

The module is trigger-agnostic — it records whatever error_kind the
caller passes. Classification of which errors are retryable (429/5xx/
network only) lives in the proxy layer (commit 6).

read-modify-write in record_failure is intentionally non-atomic; under
single-process uvicorn the worst case is one lost doubling step, which
the next failure re-escalates. Noted in a code comment.

list_active (until_ts DESC) backs the admin backoff panel; clear_provider
and clear_all back the admin "reset backoff" actions. tests/test_backoff.py
covers the curve (initial, doubling, cap reached+held, natural-expiry
reset, exact-expiry boundary), success clear + counter reset, get/is_backed_off
expired filtering, clear scope, list ordering, and pair independence.
A pure sqlite-backed store for the last-success (provider, key_name) per
(client_key_hash, canonical_model) pair, per docs/requirements-sticky-backoff.md
§2. Read at request entry so the proxy can try the cached-warm target first;
upserted on every success so warmth survives restarts.

Three functions:
- get(db, client, model) -> dict|None  (read by composite PK; no expiry)
- set_on_success(db, client, model, provider, key, *, now=None)  (upsert;
  ON CONFLICT updates provider/key_name/last_success_ts)
- list_all(db) -> list[dict]  (ordered by last_success_ts desc, for admin §7)

The module is a pure store: config-validity, backoff interaction, and the
'sticky moves only on a different candidate's success' rule (§6) live in the
routing layer (commit 5). Stickies do not expire; stale rows (referenced
key removed from config) are ignored by the routing layer and lazily
overwritten on the next success. No clear/clear_all functions — §7 marks
the clear-sticky admin action as OPEN with lean=defer.

Tests: absent->None, set->get round-trip, upsert replaces (not duplicates),
distinct pairs independent, list ordering, now-omitted uses real time, and
direct-INSERT PK enforcement.
Rewrite select_sequence to incorporate three cache-warmth mechanisms from
docs/requirements-sticky-backoff.md:

- **Sticky (§2):** the last (provider, key) that succeeded for a
  (client, model) pair is tried first, preserving upstream prompt-cache
  warmth. Read from sqlite at request entry; the proxy upserts on success.
- **Backoff filtering (§3/§4 step 6):** (provider, key) pairs currently in
  backoff are excluded from the candidate sequence for this request.
- **Cache-warmth thresholds (§6):** two knobs decide when a sticky choice
  may hold despite a priority gap. provider_priority_cache_threshold
  (across providers) and key_priority_cache_threshold (within a provider).
  Both default 0 (no warm-cache leeway).

The v1 ordering algorithm is extracted into _non_sticky_ordering (pure, no
DB). select_sequence wraps it: compute non-sticky order → backoff-filter →
if sticky exists and is valid and not backed off, apply §6 thresholds to
decide position 0 → reorder. Returns [] when all candidates are in backoff
(the proxy distinguishes aor_all_backed_off from aor_no_providers).

Config additions: provider_priority_cache_threshold,
key_priority_cache_threshold, max_opaque_retries on [routing]; per-model
max_opaque_retries override on [models.<canonical>]. The proxy call site
is updated to pass db_path + thresholds; the full walk logic with
max_opaque_retries budget and terminal 429 handling follows in commit 6.

Tests: 13 new routing tests (backoff filtering, sticky first/skipped/stale,
provider/key threshold boundaries and composition, §6.3 backed-off p_min)
+ 2 new config tests (max_opaque_retries override, global default).
Integrate the new routing state into the proxy walk:

- On a successful attempt: upsert the per-(client,model) sticky target so
  the same (provider,key) is tried first next time (spec §2), and drop any
  backoff row for that pair so the failure counter resets (spec §3).
- On a retryable failure: record a backoff row only for trigger errors
  (429/5xx/network). 402/quota-403 remain retryable for walking but do NOT
  trigger backoff — they are billing/account errors, not transient capacity
  issues (spec §3).
- Replace the v1 'max_attempts = len(sequence) if auto_fallback else 1'
  with the per-model max_opaque_retries budget (None = unlimited walk).
  Only real upstream attempts count; backoff-skipped candidates never
  become an attempt_no. Walk continues iff auto_fallback AND retryable AND
  more candidates remain AND the budget is not exhausted (spec §9.1/§10).
- Terminal: an empty filtered sequence now returns a router-generated 429
  aor_all_backed_off (config validation guarantees every catalog model has
  a serving provider, so empty == all backed off), replacing the old 503
  aor_no_providers.

sticky.set_on_success, backoff.clear, and backoff.record_failure are all
independent of auto_fallback — only the intra-request walk is gated by it,
per §9.1.

Tests: 10 new proxy integration tests covering sticky write, 429/5xx/402
backoff trigger semantics, success-clears-backoff, the aor_all_backed_off
terminal, max_opaque_retries=0 first-attempt-only, backoff filtering a
provider on the next request, and backoff recording under
auto_fallback=false.
Expose the new per-(provider,key) backoff state in the admin UI (spec §7):

- Per-provider detail view gains a 'Backoff' panel listing each currently-
  backed-off (provider, key) row with the friendly key name, a relative
  'until' time, the current window length, the failure level, and the last
  error kind. A 'Reset backoff for <provider>' button POSTs to clear that
  provider's rows (DELETE FROM backoff WHERE provider = ?).
- Overview gains a 'Backoff' summary listing all active rows with a 'Clear
  all backoffs' button (DELETE FROM backoff).
- Both resets are POST forms gated by the admin session, redirect back to
  the stats view, and flash a one-line confirmation via a query param.
- A new relative_until helper renders the future expiry compactly ('in 5s',
  'in 2m', 'in 1h').

The backoff panel keys its display by key_name (the stable [api_keys.<name>]
identity the backoff table stores), not env, via a new _key_display_names
map; an unset friendly name falls back to the key_name identifier.

Deferred per spec: the active-stickies panel (optional) and a clear-sticky
action (OPEN, lean defer) are not added; the sticky store is self-healing
under normal operation.

Tests: 8 new admin tests covering the detail panel (populated/empty), the
overview summary (count/empty), per-provider reset (clears + flash + scope),
clear-all (clears + flash), and session-gating of both reset routes.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
devtrev/actually-open-router!1
No description provided.