How we think about retries in multi-agent systems

I spent four years of a PhD making large models do small things correctly, which mostly meant getting very familiar with every way a model call can fail. The thing that surprised me moving from research into building a production framework is how often "retry logic" gets treated as a single, undifferentiated feature — as if every failure deserves the same response. It doesn't, and treating them the same wastes money, time, or both.

Four kinds of failure, four different shapes

Strip away the specific error message and almost every LLM call failure falls into one of four buckets:

• Transient errors. A dropped connection, a 503, a provider having a bad thirty seconds. Retrying after a short delay usually just works.
• Rate limits.Structurally similar to a transient error, but retrying immediately makes it worse, not better — you're adding load to a limit you just hit.
• Semantic failures.The call succeeded at the network layer, but the model's response doesn't parse, or doesn't validate against the schema you declared. This is a completely different failure mode from the first two — nothing about the network was wrong.
• Timeouts.The call is taking too long, for reasons that could be provider load, an unusually large prompt, or a model that's decided to think much longer than you budgeted for.

A retry strategy that doesn't distinguish between these ends up either too aggressive (hammering a rate limit) or too passive (giving up on a semantic failure that a fresh sample would have fixed on the second try).

Why exponential backoff with jitter, specifically

n00dles' default retry policy is exponential backoff with jitter, configurable per agent:

@agent(model="claude-sonnet-4-6", retry=5)
def flaky_call(x: str) -> str:
    """..."""
# backoff: ~1s, ~2s, ~4s, ~8s, ~16s (± jitter), then raises

Exponential growth handles the rate-limit case well almost by construction — by the third or fourth attempt, you're waiting long enough that you're not just retrying into the same limit window. The jitter matters for a reason that's easy to miss if you're only testing with one pipeline run at a time: if a provider has a brief outage and a hundred of your pipeline runs all hit a transient error within the same second, fixed-interval backoff means all hundred retry at the exact same moment again. That's a self-inflicted thundering herd, aimed at a provider that's already having a bad time. Jitter spreads those retries out so recovery doesn't create its own spike.

Semantic failures need the same retry, a different reason

Here's the part that took us longest to get right. When an agent declares a structured return type and the model's response doesn't validate against it, n00dles raises AgentOutputError— and that error goes through the exact same retry policy as a network failure. That's deliberate: LLM output is stochastic, so a response that failed to parse on one sample frequently succeeds on the next, for the same reason re-rolling a slightly ambiguous prompt sometimes just works. Treating a semantic failure as categorically different from a network failure would mean writing two retry code paths that do almost the same thing, for marginal benefit.

When retries run out: fallback agents

Sometimes the right answer after exhausting a retry budget isn't to fail the whole pipeline — it's to hand the job to a different agent. A fallback=agent runs once, after the primary agent's retries are exhausted, and its result is checkpointed exactly like a normal success:

@agent(model="gpt-4o-mini")
def cheap_backup(x: str) -> str:
    """..."""

@agent(model="claude-sonnet-4-6", retry=3, fallback=cheap_backup)
def primary(x: str) -> str:
    """..."""

This is particularly useful when the primary failure mode is provider-specific — if Anthropic is having a rough afternoon, falling back to an OpenAI model for that one node is often a better outcome than failing the whole run and forcing a manual retry later.

What retry doesn't solve, and what does

Retry and fallback both operate within a single run — they answer "what do we do right now, given this call just failed." They don't answer a related but different question: "a provider has been failing consistently for the last five minutes across hundreds of runs — should we stop sending it traffic at all for a while?" That's what a circuit breaker is for, and it isn't shipped yet.

We'd rather tell you exactly where that line is than let the roadmap blur into the feature list. See the error handling docsfor the full retry/fallback reference and the current state of what's shipped versus planned.