← All cheatsheets

AI CLI

AI SRE — operating LLMs and agents in production

Run LLM and agent apps like production systems: watch the four AI golden signals (quality, latency, cost, errors), trace a multi-step agent run, hold the line with budgets, timeouts, retries, fallback models and circuit breakers, gate every prompt/model change on an eval, and work an AI incident where a change regressed quality or spiked cost.

Docs-checkedai-srellmopsobservabilityreliabilitycost

Quick reference — reach for these first

1. Which signal fired?open dashboard: quality / latency / cost / errorsName the golden signal and the blast radius (one route? one model? all traffic?) before touching anything.
2. Pull the eval trendeval history --last 7d # pass-rate over timeIs quality actually down, and since when? The breakpoint timestamp points at the change that caused it.
3. Diff the recent changegit log -p --since=<breakpoint> -- prompts/ config/Correlate the breakpoint with a prompt edit, model-id bump, temperature change, or tool/deploy. That is your suspect.
4. Read failing tracestraces --filter 'score<1' --limit 5 # input+output+toolsOpen 3-5 bad runs end to end. See the real input, the bad output, token counts, and which hop/tool misfired.
5. Mitigate: roll back or routerevert prompt | route to fallback model | lower max_tokensStop the bleeding first (revert/fallback/cap), then re-run the eval gate to confirm recovery before you root-cause.

Four AI golden signals feed an alert; the request path emits them at every hop

AI golden signals and the request pathThe request path client to guardrail to model and tools to eval emits telemetry; four golden signals quality, latency, cost and errors feed one alert that pages on-call when a threshold is crossed.ClientGuardrailinput validationModel + Toolsmulti-step runEvalonline scoreUserQualityeval pass-rateLatencyp95 + TTFTCosttok + $ / callErrorstool / 5xx / 429 / refusalAlertthreshold crossed -> page on-calleach hop emits telemetry

Text equivalent: Two linked views. Top: the request path runs left to right — Client to Guardrail (input validation) to Model and Tools to Eval (online scoring) to the user. Each hop emits telemetry downward. Bottom: four golden signals — Quality (eval pass-rate), Latency (p95 incl. time-to-first-token), Cost (tokens and dollars per call), and Errors (tool, 5xx, 429, refusal rate) — all feed into a single Alert box. When any signal crosses its threshold, the alert pages on-call.

Common tasks

Instrument · Protect · Respond

InstrumentProtectRespond
Define the four AI golden signals and an SLO for each# quality=eval pass-rate, latency=p95 incl. TTFT, cost=$/1k calls, errors=tool+5xx+429 rateEach signal has a target number and a page/alert threshold written down, derived from a baseline run.
Emit a trace per request with a shared run/trace idlog {trace_id, route, model, in_tok, out_tok, ttft_ms, latency_ms, tool, score}One prod request produces one trace you can open and read every hop of, keyed by trace_id.
Capture time-to-first-token separately from total latencystream=True; record ttft at first chunk, total at lastDashboard shows p50/p95 TTFT and total latency as two distinct lines, not one blended number.
Meter token cost per request and roll up to $/1k callscost = in_tok*in_price + out_tok*out_price # pin provider pricesA cost-per-request and daily-spend panel exists; a 2x token blowup is visible within minutes.
Set a timeout and a hard max_tokens on every model callclient.messages.create(..., max_tokens=N, timeout=20)No call can hang unbounded or emit unbounded output; a runaway generation is capped, not infinite.
Retry transient failures with exponential backoff + jitterretry on 429/503/timeout: sleep(base*2**n + rand); cap attempts=3A transient 429 recovers automatically, and retries are capped so they cannot become a retry storm.
Add a fallback model and a circuit breakeron repeated errors: open breaker -> route to fallback / cached / degraded replyWhen the primary model errors or times out, traffic degrades gracefully instead of hard-failing users.
Cache reusable prompt prefixes and identical requestsenable prompt caching on the static system/context prefixRepeat/expensive calls show cache hits; cost and latency drop measurably on cached paths.
Guard inputs and outputs (PII, injection, schema)pre: strip/flag PII + injection; post: validate JSON/schema + policyA prompt-injection or malformed-output test case is blocked and logged, not passed through to the user.
Cap agent steps and tool calls to kill runaway loopsmax_steps=N; per-run tool-call budget; stop on repeated identical actionAn agent that loops hits the step/tool ceiling and halts with an error, instead of billing forever.
Gate every prompt/model change on an eval in CIeval run candidate; diff vs baseline; block merge on any regressionA prompt edit that drops the pass-rate fails CI and cannot merge; see ai-output-evaluation.
Run online eval: sample live traffic and score it continuouslysample X% of prod; async judge/rubric; alert on score driftA silent quality drop in prod raises an alert from the online score, not from a user complaint.
Write the AI runbook: who pages, how to roll back, how to verifyrunbook: signal -> suspect -> revert/fallback -> re-eval -> all-clearOn-call can roll back a prompt/model and confirm recovery from the runbook without tribal knowledge.

Quality dropped after a prompt change — triage, roll back, and verify with the eval gate

AI incident response

1
The quality signal paged: online eval pass-rate fell. Pull the trend and find the breakpoint.
eval history --route support-summarizer --last 7d
# pass-rate 0.94 -> 0.94 -> 0.78 starting 2026-11-13 15:40Z
# break lines up with a deploy at 15:38Z
2
Diff what shipped at the breakpoint. A prompt edit is the suspect (model id unchanged).
git log -p --since='2026-11-13 15:00Z' -- prompts/
# - "Summarize the ticket in 3 bullet points."
# + "Summarize the ticket as a short paragraph."
# model id + temperature unchanged; prompt is the only change
3
Open 3-5 failing traces to see the SAMPLE INPUT and the bad output the change produced.
traces --route support-summarizer --filter 'score<1' --limit 5
# SAMPLE INPUT: "Customer can't log in after password reset; tried twice; using iOS app v4.2."
# BAD OUTPUT (new prompt): a 180-word paragraph; downstream router expected 3 bullets -> parse fail -> score=0
# regression is structural: the new wording broke the format the next step depends on
4
Mitigate first: revert the prompt to stop the bleeding (COPY-READY restored prompt).
# prompts/support-summarizer.txt (restored)
Summarize the ticket in exactly 3 bullet points.
Each bullet: one sentence, under 20 words.

git revert <deploy-sha> && deploy
# reverted; traffic now serves the known-good 3-bullet prompt
5
EVALUATE the fix: re-run the eval gate against the frozen golden set and diff vs baseline (EXPECTED: score recovers).
eval run --prompt prompts/support-summarizer.txt --set golden.jsonl > after.txt
diff baseline.txt after.txt
# pass-rate 0.78 -> 0.94 (matches baseline)
# format-valid 0.75 -> 1.00  (bullets parse again)
# DECISION: recovered to baseline -> all-clear
6
Confirm the roll back did not trade quality for cost or latency before closing the incident.
signals --route support-summarizer --compare pre-incident
# tokens/call ~ flat, p95 latency ~ flat, $/1k calls ~ flat
# all four golden signals back to baseline -> close incident, add the broken-format case to golden.jsonl

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Cost spiked suddenly (token blowup, retry storm, or lost cache)Break daily spend down by route and model; chart tokens-per-request and retry count over the spike window.Whether it is more/longer outputs (bigger context, no max_tokens), a retry storm on errors, or prompt-cache hit-rate collapsing.Cap max_tokens and context, add/fix prompt caching, cap retry attempts with backoff, and route cheap traffic to a smaller model.
Latency up, especially p95 / time-to-first-tokenSplit TTFT vs total latency and attribute by hop: model size, prompt/context length, and number of sequential tool calls.Whether a model/id bump, a longer prompt or context, or added tool round-trips is the cause.Stream for TTFT, trim context, parallelize independent tool calls, cache the static prefix, or route to a faster model for latency-sensitive routes.
Quality regressed silently — users complain, no alert firedPull the online/offline eval score trend; if flat or missing, you have no quality signal wired.That there was no eval gate or online eval catching the drop — the regression escaped because nothing measured it.Add an eval gate in CI on every prompt/model change and sample+score live traffic; add each escaped failure as a permanent regression case.
Guardrail false-positives — valid requests get blocked or refusedPull the guardrail/refusal rate and read a sample of blocked inputs against the rule that fired.Whether the input filter or output policy is too aggressive, versus a genuine spike in abusive traffic.Tighten the rule to the real threat, add allow-list cases to the guardrail's own eval, and alert on refusal-rate drift so overblocking is visible.
Agent in an infinite loop / runaway tool callsOpen one offending trace and count steps and repeated identical tool actions before the run ballooned.Whether the agent lacks a step/tool budget or a stop condition and is re-trying the same failing action.Enforce max_steps and a per-run tool-call budget, halt on repeated identical actions, and add a timeout so a stuck run is killed, not billed.
Error/refusal rate climbing (429s, tool errors, 5xx)Break the error rate by type — rate-limit (429) vs tool/API error vs model refusal — and by route.Whether you are hitting provider rate limits, a downstream tool is failing, or a prompt change raised refusals.Back off and smooth traffic for 429s, add retries+fallback for tool errors, and revert the prompt if refusals jumped after a change.
Cannot tell which hop of an agent run failedConfirm every model and tool call shares one trace_id and logs its own input, output, tokens, and latency.Whether tracing is actually end-to-end or stitched from disconnected logs you cannot correlate.Propagate a single trace_id through every step and emit a span per hop (an OpenTelemetry GenAI-style trace) so a run reads top to bottom.

References

Signal names and patterns are provider-agnostic; examples reference a current Claude Sonnet/Opus 4.x id generically — confirm the exact model id, token prices, and rate limits in the Anthropic docs before wiring cost/latency math or alert thresholds. Thresholds (p95 latency, $/1k calls, pass-rate floor) are placeholders you set from your own SLOs and a baseline run, not universal constants. Eval quality gating assumes you already have a frozen golden set and a scorer (see ai-output-evaluation); this sheet is the operating layer on top of it.