Two CI retries turn a 60% agent into a green build
Retry a failed agent test twice and your CI gate stops asking whether the agent works and starts asking whether it can work once in three tries. The arithmetic, and a fix borrowed from factory inspection.
If your pipeline retries a failed test twice before calling it red, an agent that completes a task six times in ten will pass that gate about 94% of the time. The retry setting didn't make the agent better. It changed the question the gate is asking, from "does this work" to "can this work at least once in three tries", and nobody wrote that decision down.
The rest of this is arithmetic, plus one idea I borrowed from factory inspection.
Why is regression testing a non-deterministic agent different from fixing a flaky test?
Flaky tests are an old problem with an old answer. A test that fails sometimes without a code change is treated as a defect in the test: a race, a shared fixture, a slow selector. You retry it to keep the build moving, you quarantine it if it keeps flapping, and somebody is supposed to fix it.
Playwright's documentation puts the convention plainly. A test that "failed on the first run, but passed when retried" is labelled flaky, and in their own example output one flaky test sits next to two passed tests in a run that succeeds. The build goes green. That's reasonable for a login form, because the login form either works or it doesn't, and the variance really is noise in the harness.
An agent is a different thing. The variance lives in the product. Run the same task through the same agent with the same prompt and you get different tool calls, different orderings, and sometimes a different outcome. A failure on attempt one followed by a pass on attempt two tells you, directly, that the agent fails this task some fraction of the time. Calling that flaky files a measurement under noise.
Tian Pan made the key distinction back in April: "A traditional test that fails intermittently is a bug to fix. An LLM eval that fails intermittently might be signal." I agree with him, and I want to push it one step further, into the number the gate actually reports.
What does a retry policy actually measure?
Anthropic's engineering team published the vocabulary for this in January. Pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts." Pass^k "measures the probability that all k trials succeed." Their example: a 75% per-trial success rate over three trials gives (0.75)³, about 42%, for pass^3.
Now look at what "retry twice on failure" does. It runs up to three attempts and stops at the first success. The build is green if at least one attempt passes. That is pass@3, by definition. Nobody chose pass@3. It fell out of a YAML default written for login forms.
Here's what that does across a range of agents, where p is the true per-run success rate. I ran these on 27 September; the formula is 1 minus (1 minus p) cubed.
| True per-run success | Gate passes with 2 retries (pass@3) | All three pass (pass^3) |
|---|---|---|
| 95% | 99.99% | 85.7% |
| 90% | 99.9% | 72.9% |
| 80% | 99.2% | 51.2% |
| 70% | 97.3% | 34.3% |
| 60% | 93.6% | 21.6% |
| 50% | 87.5% | 12.5% |
Read the middle column top to bottom. An agent that falls from 90% to 70% per run has lost a fifth of its reliability, and the gate's pass probability moves from 99.9% to 97.3%. On any single pull request you will not see that. You'd need dozens of builds before the extra reds became noticeable, and by then each one looks like the flake it's labelled as.
The same Anthropic post says regression evals "should have a nearly 100% pass rate". With retries on, they will have one. That's the problem.
Hasn't someone already written this?
My first thought was that this is too obvious to be unwritten. So I went looking before drafting.
A lot of it is taken, and I'd rather say so. Anthropic defined the metrics. Tian Pan argued intermittent eval failure is signal and that single runs are "point estimates, not distributions". Saurav Bhattacharya showed on dev.to in June that the judge wobbles too: 200 frozen responses through the same judge produced 14 verdict flips, and his line "A 95% pass rate at 0.6 judge agreement is noise" is better than anything I'd write on that half. Varun Pratap Bhardwaj's AgentAssay preprint from March proposes three-valued verdicts (PASS, FAIL, INCONCLUSIVE), sequential testing and adaptive trial budgets, and reports a 78% reduction in trials using a sequential probability ratio test. That's the rigorous version of where this piece ends up.
What I couldn't find anyone saying directly is the narrower thing above: that the retry key in a CI config is itself a metric choice, that it selects pass@k, and that the table shows how blind it is. Steve Kinney's course has a good section on retries hiding real flakes in ordinary suites, but it's about app tests, not agents, and it does no arithmetic. So the claim I'm making is small. I think it's also the one a tired engineer needs at 6pm when the agent test goes red once and green on rerun.
What would a factory inspector do with this?
This is the part that made it click for me, and it comes from somewhere with no LLMs in it.
Manufacturing solved "we can't test everything and results vary" decades ago with acceptance sampling. The NIST/SEMATECH engineering statistics handbook describes the basic plan like this: one sample is drawn at random from a lot, and the plans "are usually denoted as (n,c) plans for a sample size n, where the lot is rejected if there are more than c defectives."
So a plan is two numbers you set on purpose: how many to look at, and how many failures you'll tolerate.
Write a CI retry policy in those terms. Retries of 2 means n = 3, and the lot is rejected only if all three are defective. That's c = 2 out of 3. No quality engineer would sign off on a plan that accepts a lot when two of three sampled parts are bad. We do it every day in agent pipelines and call it resilience.
The inspector's version for an agent gate looks like this. Pick the per-run rate you consider healthy, pick the rate you consider a regression, and choose n and c so the healthy agent usually passes and the regressed one usually fails. For a healthy 90% and a regressed 70%, I worked out these single-sample plans on 27 September:
| Runs (n) | Pass if at least | Healthy 90% agent passes | Regressed 70% agent passes |
|---|---|---|---|
| 5 | 3 | 99.1% | 83.7% |
| 10 | 7 | 98.7% | 65.0% |
| 20 | 16 | 95.7% | 23.8% |
| 50 | 41 | 97.5% | 4.0% |
Five runs barely separates them. Twenty runs catches the regression about three times in four. Fifty catches it nearly always. Every row is a plan somebody chose, with its miss rate visible, which is the thing a retry setting hides.
Isn't twenty runs per pull request too expensive?
Sometimes, yes. This is where I'm guessing rather than knowing.
You don't need the plan on every test, only on the tasks where a regression would hurt, the ones that write, pay, delete or send. And sequential testing stops early once the answer is clear, which is what AgentAssay's 78% trial reduction is measuring on their setup.
The comparison is also rigged in favour of retries. The cheap option isn't free. It's a gate that reports green for a 70% agent, and you pay for that later in production, where a user gets one attempt and experiences something much closer to pass@1 than pass@3.
And one habit to drop entirely: quarantine. Quarantining a flaky login test is sensible housekeeping. Quarantining an agent test switches off the only sensor pointed at that behaviour. If an agent task is too variable to gate, that's a finding about the agent, and it belongs in the release notes, not in a skipped-tests list.
What I'm confident of, and what I'm not
The arithmetic and the definitions are established; you can check both tables with a calculator. That most teams gating agents in CI are running retry settings inherited from ordinary test suites is inferred from how CI defaults and flaky-test practice work, not from a survey. The trial counts are my guess at what a team will actually pay for, and I'd change them the moment someone shows me real cost data.
The claim, in one sentence: a CI retry policy is a decision about which metric you gate on, and for an agent the usual retry setting gates on the most forgiving metric there is.
This is the third piece in a run about agents in production. The MCP schema drift piece was about a contract changing with nothing in the pipeline failing, and the collusion piece was about a checker that stops checking while the output stays good. This is the same family: a gate that keeps reporting green because of how it counts, not because of what it saw.
Sources
Anthropic Engineering, Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe, "Demystifying evals for AI agents", 9 January 2026. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
Playwright documentation, "Retries". https://playwright.dev/docs/test-retries
Varun Pratap Bhardwaj, "AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows", arXiv:2603.02601, 3 March 2026. https://arxiv.org/abs/2603.02601
Tian Pan, "AI Agents in Your CI Pipeline: How to Gate Deployments That Can't Be Unit Tested", 17 April 2026. https://tianpan.co/blog/2026/04/17/ai-agents-ci-pipeline-deployment-gates
Saurav Bhattacharya, "Your Evals Are Flaky Too: Stop Trusting a Pass Rate You Can't Reproduce", dev.to, 25 June 2026. https://dev.to/saurav_bhattacharya/your-evals-are-flaky-too-stop-trusting-a-pass-rate-you-cant-reproduce-6pk
Steve Kinney, "Flaky-Test Triage: When Retries Are Lying to You". https://stevekinney.com/courses/self-testing-ai-agents/flaky-test-triage
NIST/SEMATECH e-Handbook of Statistical Methods, sections 6.2.1 "What is Acceptance Sampling?" and 6.2.2 "What kinds of Lot Acceptance Sampling Plans (LASPs) are there?" https://www.itl.nist.gov/div898/handbook/pmc/section2/pmc22.htm
Calculations: binomial probabilities computed 27 September 2026. Retry table uses 1 minus (1 minus p) cubed, and p cubed. Sampling table uses the binomial probability of at least c successes in n trials.
