AI CLI
AI SRE — operating LLMs and agents in production
Run LLM and agent apps like production systems: watch the four AI golden signals (quality, latency, cost, errors), trace a multi-step agent run, hold the line with budgets, timeouts, retries, fallback models and circuit breakers, gate every prompt/model change on an eval, and work an AI incident where a change regressed quality or spiked cost.
Quick reference — reach for these first
open dashboard: quality / latency / cost / errorsName the golden signal and the blast radius (one route? one model? all traffic?) before touching anything.eval history --last 7d # pass-rate over timeIs quality actually down, and since when? The breakpoint timestamp points at the change that caused it.git log -p --since=<breakpoint> -- prompts/ config/Correlate the breakpoint with a prompt edit, model-id bump, temperature change, or tool/deploy. That is your suspect.traces --filter 'score<1' --limit 5 # input+output+toolsOpen 3-5 bad runs end to end. See the real input, the bad output, token counts, and which hop/tool misfired.revert prompt | route to fallback model | lower max_tokensStop the bleeding first (revert/fallback/cap), then re-run the eval gate to confirm recovery before you root-cause.Four AI golden signals feed an alert; the request path emits them at every hop
Text equivalent: Two linked views. Top: the request path runs left to right — Client to Guardrail (input validation) to Model and Tools to Eval (online scoring) to the user. Each hop emits telemetry downward. Bottom: four golden signals — Quality (eval pass-rate), Latency (p95 incl. time-to-first-token), Cost (tokens and dollars per call), and Errors (tool, 5xx, 429, refusal rate) — all feed into a single Alert box. When any signal crosses its threshold, the alert pages on-call.
Common tasks
Instrument · Protect · Respond
| Instrument | Protect | Respond |
|---|---|---|
| Define the four AI golden signals and an SLO for each | # quality=eval pass-rate, latency=p95 incl. TTFT, cost=$/1k calls, errors=tool+5xx+429 rate | Each signal has a target number and a page/alert threshold written down, derived from a baseline run. |
| Emit a trace per request with a shared run/trace id | log {trace_id, route, model, in_tok, out_tok, ttft_ms, latency_ms, tool, score} | One prod request produces one trace you can open and read every hop of, keyed by trace_id. |
| Capture time-to-first-token separately from total latency | stream=True; record ttft at first chunk, total at last | Dashboard shows p50/p95 TTFT and total latency as two distinct lines, not one blended number. |
| Meter token cost per request and roll up to $/1k calls | cost = in_tok*in_price + out_tok*out_price # pin provider prices | A cost-per-request and daily-spend panel exists; a 2x token blowup is visible within minutes. |
| Set a timeout and a hard max_tokens on every model call | client.messages.create(..., max_tokens=N, timeout=20) | No call can hang unbounded or emit unbounded output; a runaway generation is capped, not infinite. |
| Retry transient failures with exponential backoff + jitter | retry on 429/503/timeout: sleep(base*2**n + rand); cap attempts=3 | A transient 429 recovers automatically, and retries are capped so they cannot become a retry storm. |
| Add a fallback model and a circuit breaker | on repeated errors: open breaker -> route to fallback / cached / degraded reply | When the primary model errors or times out, traffic degrades gracefully instead of hard-failing users. |
| Cache reusable prompt prefixes and identical requests | enable prompt caching on the static system/context prefix | Repeat/expensive calls show cache hits; cost and latency drop measurably on cached paths. |
| Guard inputs and outputs (PII, injection, schema) | pre: strip/flag PII + injection; post: validate JSON/schema + policy | A prompt-injection or malformed-output test case is blocked and logged, not passed through to the user. |
| Cap agent steps and tool calls to kill runaway loops | max_steps=N; per-run tool-call budget; stop on repeated identical action | An agent that loops hits the step/tool ceiling and halts with an error, instead of billing forever. |
| Gate every prompt/model change on an eval in CI | eval run candidate; diff vs baseline; block merge on any regression | A prompt edit that drops the pass-rate fails CI and cannot merge; see ai-output-evaluation. |
| Run online eval: sample live traffic and score it continuously | sample X% of prod; async judge/rubric; alert on score drift | A silent quality drop in prod raises an alert from the online score, not from a user complaint. |
| Write the AI runbook: who pages, how to roll back, how to verify | runbook: signal -> suspect -> revert/fallback -> re-eval -> all-clear | On-call can roll back a prompt/model and confirm recovery from the runbook without tribal knowledge. |
Quality dropped after a prompt change — triage, roll back, and verify with the eval gate
AI incident response
eval history --route support-summarizer --last 7d
# pass-rate 0.94 -> 0.94 -> 0.78 starting 2026-11-13 15:40Z
# break lines up with a deploy at 15:38Zgit log -p --since='2026-11-13 15:00Z' -- prompts/
# - "Summarize the ticket in 3 bullet points."
# + "Summarize the ticket as a short paragraph."
# model id + temperature unchanged; prompt is the only changetraces --route support-summarizer --filter 'score<1' --limit 5
# SAMPLE INPUT: "Customer can't log in after password reset; tried twice; using iOS app v4.2."
# BAD OUTPUT (new prompt): a 180-word paragraph; downstream router expected 3 bullets -> parse fail -> score=0
# regression is structural: the new wording broke the format the next step depends on# prompts/support-summarizer.txt (restored)
Summarize the ticket in exactly 3 bullet points.
Each bullet: one sentence, under 20 words.
git revert <deploy-sha> && deploy
# reverted; traffic now serves the known-good 3-bullet prompteval run --prompt prompts/support-summarizer.txt --set golden.jsonl > after.txt
diff baseline.txt after.txt
# pass-rate 0.78 -> 0.94 (matches baseline)
# format-valid 0.75 -> 1.00 (bullets parse again)
# DECISION: recovered to baseline -> all-clearsignals --route support-summarizer --compare pre-incident
# tokens/call ~ flat, p95 latency ~ flat, $/1k calls ~ flat
# all four golden signals back to baseline -> close incident, add the broken-format case to golden.jsonlTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Cost spiked suddenly (token blowup, retry storm, or lost cache) | Break daily spend down by route and model; chart tokens-per-request and retry count over the spike window. | Whether it is more/longer outputs (bigger context, no max_tokens), a retry storm on errors, or prompt-cache hit-rate collapsing. | Cap max_tokens and context, add/fix prompt caching, cap retry attempts with backoff, and route cheap traffic to a smaller model. |
| Latency up, especially p95 / time-to-first-token | Split TTFT vs total latency and attribute by hop: model size, prompt/context length, and number of sequential tool calls. | Whether a model/id bump, a longer prompt or context, or added tool round-trips is the cause. | Stream for TTFT, trim context, parallelize independent tool calls, cache the static prefix, or route to a faster model for latency-sensitive routes. |
| Quality regressed silently — users complain, no alert fired | Pull the online/offline eval score trend; if flat or missing, you have no quality signal wired. | That there was no eval gate or online eval catching the drop — the regression escaped because nothing measured it. | Add an eval gate in CI on every prompt/model change and sample+score live traffic; add each escaped failure as a permanent regression case. |
| Guardrail false-positives — valid requests get blocked or refused | Pull the guardrail/refusal rate and read a sample of blocked inputs against the rule that fired. | Whether the input filter or output policy is too aggressive, versus a genuine spike in abusive traffic. | Tighten the rule to the real threat, add allow-list cases to the guardrail's own eval, and alert on refusal-rate drift so overblocking is visible. |
| Agent in an infinite loop / runaway tool calls | Open one offending trace and count steps and repeated identical tool actions before the run ballooned. | Whether the agent lacks a step/tool budget or a stop condition and is re-trying the same failing action. | Enforce max_steps and a per-run tool-call budget, halt on repeated identical actions, and add a timeout so a stuck run is killed, not billed. |
| Error/refusal rate climbing (429s, tool errors, 5xx) | Break the error rate by type — rate-limit (429) vs tool/API error vs model refusal — and by route. | Whether you are hitting provider rate limits, a downstream tool is failing, or a prompt change raised refusals. | Back off and smooth traffic for 429s, add retries+fallback for tool errors, and revert the prompt if refusals jumped after a change. |
| Cannot tell which hop of an agent run failed | Confirm every model and tool call shares one trace_id and logs its own input, output, tokens, and latency. | Whether tracing is actually end-to-end or stitched from disconnected logs you cannot correlate. | Propagate a single trace_id through every step and emit a span per hop (an OpenTelemetry GenAI-style trace) so a run reads top to bottom. |
References
- Google SRE Book — Monitoring Distributed Systems (the golden signals)sre.google
- Anthropic — Reduce latencydocs.anthropic.com
- Anthropic — Prompt cachingdocs.anthropic.com
- Anthropic — Errors and rate limitsdocs.anthropic.com
- OpenTelemetry — Generative AI semantic conventions (traces/metrics)opentelemetry.io
Signal names and patterns are provider-agnostic; examples reference a current Claude Sonnet/Opus 4.x id generically — confirm the exact model id, token prices, and rate limits in the Anthropic docs before wiring cost/latency math or alert thresholds. Thresholds (p95 latency, $/1k calls, pass-rate floor) are placeholders you set from your own SLOs and a baseline run, not universal constants. Eval quality gating assumes you already have a frozen golden set and a scorer (see ai-output-evaluation); this sheet is the operating layer on top of it.