Why AI Agents Fail on Long-Running Tasks

The demo always works. The demo is four steps long. Every operations leader who has piloted an AI agent has seen the same arc: a task the agent nails in a meeting, and the same task falling over when it runs forty steps deep against real data on a Tuesday morning. That gap is not bad luck and it is not a prompt you have yet to write. It is the single most consistent finding in the agent research literature, and it has a shape you can calculate.

The benchmarks are much worse than the demos

The most honest measurement of agents doing real professional work is TheAgentCompany, a Carnegie Mellon benchmark that drops an agent into a simulated software company: a real intranet, a real code host, real colleagues to message, and 175 long-horizon tasks spanning engineering, project management, data science, administration, HR, and finance. Not puzzles. The work.

In the published results the strongest model completed 30.3% of tasks autonomously, and scored 39.3% on the metric that hands out partial credit for getting most of the way. The best results reported since sit in the low forties. That is the ceiling, against a benchmark whose tasks a competent new hire would clear.

The category split is the part worth sitting with. Agents do best on software engineering, which is the domain they were trained hardest on and the domain where a failed attempt costs nothing. They do worst on finance and HR: work that is not intellectually harder, but that is longer, spans more systems, and depends on state that has to survive from the first step to the last. Long is what breaks the loop.

Reliability does not average. It multiplies.

Here is the whole problem in one line of arithmetic. If each step of a workflow succeeds independently with probability p, then a workflow of n steps finishes clean with probability pⁿ. Not p. Not something near p. The reliability you measured on one step, raised to a power.

A step that works 95 times out of 100 sounds like a step you can build on. Chain twenty of them and the workflow finishes clean 36% of the time. Push per-step reliability all the way to 99% and a hundred-step process still lands at 37%.

This is where the practical rule of thumb comes from. Ask an agent to do three things and it usually holds. Ask it to do ten and you are rolling dice you cannot see. The failure is not that the agent is bad at any one step, it is that you are asking it to be right ten times in a row and pricing it as though you asked once.

Toby Ord, at the Oxford Martin AI Governance Initiative, showed that agent success rates fit a constant hazard rate: a steady chance of failing during each minute of work, which produces exponential decay and gives every model something like a half-life, in the same sense as radioactive decay. The consequence is blunt. Double the length of a task and you square the probability of success. An agent with a 50% chance on a one-hour task has a 25% chance on a two-hour task and a 6% chance on a four-hour one.

The degradation starts at step two

Compounding arithmetic assumes each step is independent. In a real agent loop they are not, and the dependence runs the wrong way.

Researchers at Microsoft Research and Salesforce ran 200,000 simulated conversations delivering the same task two ways: all at once, or spread across turns. Spreading it out cost an average of 39% of performance, in every model tested, from small open-weight models to the frontier. The drop appeared at two turns. Their summary of the mechanism is the line worth remembering: when models take a wrong turn, they get lost, and they do not recover.

Two things are happening underneath that, and both get worse with length:

• Self-conditioning. An agent’s own earlier output sits in its context window. When one of those outputs is wrong, the model reads it as established fact and builds the next step on top of it. One bad step does not fail in isolation, it corrupts the ground every later step stands on.
• Context rot. Chroma’s research across eighteen frontier models found accuracy degrades non-uniformly as input grows, often well before the documented limit. A 200K window can lose meaningful accuracy at 50K of input. A long-running agent fills its own context with transcript, and the fuller it gets, the worse it reasons over it.

So the picture is not an agent that works until it hits a wall. It is an agent that quietly gets worse from step two onward, with no error raised, no alert fired, and no obvious moment to point at afterwards. For a client deliverable, silent drift is the worst possible failure mode. A crash you can catch. A plausible wrong answer goes out the door.

A better model moves the half-life. It does not change the shape.

The counterargument is that this is temporary, and models are improving fast. They are. METR found the length of task a frontier agent can complete at 50% reliability has doubled roughly every seven months for six years, and the recent trend is faster still. That is a real and genuinely remarkable slope.

But read what the metric says. It is the task length at 50% reliability: a coin flip. The same research reports that the horizon at 80% reliability is roughly five times shorter than the horizon at 50%. Demanding that an agent be right four times out of five instead of one time in two cuts the length of work you can hand it by around 80%.

Now extend that to what operations actually requires. Nobody signs off on an 80% billing run. Every additional nine you ask for buys back a shorter and shorter task. Capability gains slide the curve to the right; they do not flatten it. The exponent is structural.

This is the mechanism behind the number the analysts keep quoting. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, on cost, unclear value, and inadequate risk controls. Those are the symptoms. The exponent is the disease: pilots demo beautifully at four steps and never survive contact with the forty-step version of the same job.

The fix is not a smarter agent. It is fewer stochastic steps.

If the problem is pⁿ, there are exactly two levers: raise p or lower n. The entire industry has spent three years pushing on p. The interesting results came from attacking n.

Cognizant AI Lab published a result last year in which an LLM system completed a million-step task with zero errors. The headline is not that the model got smarter. It is that the task was decomposed so each agent did exactly one step, with correction applied at every step, which turns exponential decay into a per-step error budget you can actually engineer against. Structure did what scale could not.

Caddi is that insight pointed at operations work rather than at a research benchmark. We wrote about the general pattern in what hybrid agents are. This is the reliability case for it.

How Caddi compiles the exponent away

The person who owns the work screen-shares it once. AI watches the recording and writes the automation as code that calls your systems directly by API. The fixed steps become deterministic code. The handful of genuine read-and-decide steps stay AI, and each one is scoped to that step alone.

Take a forty-step invoice reconciliation. Run as an agent loop, that is forty probabilistic decisions, each one inheriting the drift of the thirty-nine before it. Compiled by Caddi, roughly thirty-seven of those steps are deterministic code and three are bounded AI calls. The exponent goes from forty to three, and three of the four failure modes above stop applying entirely:

• Deterministic steps do not have a success rate. Same input, same output, every run, at four in the afternoon on quarter end. Across Caddi production runs this layer measures 99.99%+ reliable, and the residual is infrastructure, an API timing out, not the automation changing its mind.
• The AI calls do not share a context. Each one sees only its own scoped input. There is no transcript accumulating, so there is no self-conditioning and no context rot. A bad output at step twelve cannot condition step thirteen, because step thirteen never sees it.
• Every AI output is checked before it moves. Results are validated against your rules, and genuinely ambiguous cases route to a person for review instead of being guessed at. That is the per-step correction the million-step result depended on.
• Nothing ships unwatched. New automations run in a safe test mode against your systems first, and every API call and action is logged across every system it touches, exportable for audit.

The end-to-end numbers follow from the structure. On execution, the deterministic layer runs at 99.99%+ across Caddi production runs. On output quality, where the bounded AI steps do their work, a Caddi customer measured 98%+ extraction accuracy across 1,000+ runs in 90 days, with every field traceable to its source document and run. Those measure two different things, and both are a long way from 30%.

What this means for the work you would hand over first

Look again at the benchmark categories. Finance at 22%, HR at 18%. Those are not the hard problems in your firm. They are intake, reconciliation, onboarding, document movement, and billing prep: work a capable coordinator does without much drama, that agents fail at because it is long, spans six systems, and has to carry state from the first step to the last.

That is exactly the work worth automating, and exactly the work an agent loop is worst at. It is also the reason the reliability question is not an engineering footnote. It decides whether the thing you piloted ever becomes something the firm runs on.

Frequently asked questions