Why Your Agent Keeps Answering After Its Tools Fail

Picture an agent that answers questions about customer invoices. It has one tool, a lookup against an internal API, and a task that needs four lookups. The first two go through fine. Then the credential behind that API expires, and every call from the third one onward comes back as a 401.

What can happen next is the part that bothers me. The agent calls the tool again, and again, each time tweaking its arguments a little, as if the API had a problem with its phrasing. After a while, it gives up on the tool and writes a tidy, confident answer that covers all four customers. Nothing in that answer admits that the tool stopped working two calls in. The run just ends with a success status, and the dashboard stays green.

This article is about that quiet ending. It looks nothing like the failures we usually plan for in agentic AI systems, and I think that is exactly why it survives so long in production.

A Failure That Ends in a Success Status

Most of the talk about agents in production is about the loud failures. Gartner expects more than 40 percent of agentic AI projects to be canceled by the end of 2027, mostly because of cost, unclear business value, and weak risk controls. That is fair enough, but those are the failures that somebody eventually notices, because a huge bill or a missing business case shows up in a meeting.

An expired credential is the opposite kind of problem. Normal software has a settled answer for it, which is to throw an error, page someone, and stop. An agent has no such instinct. To a language model, the error message is just more text in its context, and if the instructions say to answer the user’s question, it will find some way to do that, with data or without it.

This is why I am not convinced that a smarter model is the fix. A smarter model is simply better at writing a convincing answer, and that makes the one built on nothing even harder to spot. What I would rather ask is what the system around the model does when a tool dies, and whether anything forces the agent to stop answering at that point.

Not Every Failure Deserves a Retry

It is tempting to treat every tool error the same way, and that is where things start to go wrong. A timeout, a dropped connection, a rate limit, or a 503 will probably work if you wait a moment and try again. A 401 or a 403 or a 404 will fail the same way a hundred times in a row, and every retry just burns tokens while the agent looks busy.

There is also a third group that I will not pretend to solve in code: the call that half worked, or the action nobody wants repeated, like a payment that may or may not have gone through. A wrapper cannot judge that. The best it can do is stop and hand the problem to a person.

The model cannot tell these groups apart, because to it they are all just error strings, so I take the decision away from it. The sorting happens in plain code before the model sees anything. Here is the function I would start from.

import loggingimport randomimport timeimport requestslog = logging.getLogger("agent.tools")# 4xx errors other than 429 will not fix themselves, so we do not retry themRETRYABLE = {429, 500, 502, 503, 504}MAX_TRIES = 3class ToolDown(Exception):def __init__(self, status):super().__init__(f"tool failed with status {status}")self.status = statusdef call_api(url, headers, params=None):for attempt in range(1, MAX_TRIES + 1):try:resp = requests.get(url, headers=headers, params=params, timeout=10)except (requests.Timeout, requests.ConnectionError):status = 0  # no response at all, treat it like a flaky 5xxelse:if resp.ok:return resp.json()status = resp.status_codeif (status and status not in RETRYABLE) or attempt == MAX_TRIES:raise ToolDown(status)# jitter so a fleet of agents does not retry in locksteptime.sleep(2 ** attempt + random.random())

A few things in there are worth a closer look. The retry count has a hard ceiling, because a flaky tool should not be able to turn into a runaway loop. Timeouts and dropped connections get handled too, since they never produce a status code and are easy to forget. The sleep has a bit of jitter, so a fleet of agents hitting the same broken service does not all retry at the same instant. And a 401 gets raised on the first attempt, because waiting around will not renew a credential. One gap I left on purpose: a 200 response with a body that is not valid JSON would raise from resp.json() and slip past the wrapper, so catch ValueError there if your tools can do that.

One more detail: some servers send a Retry-After header with a 429, and honoring it beats a fixed backoff. It can be a date as well as several seconds though, so parse it with some care.

Teaching the Agent to Say “I Could Not”

Capping retries stops the storm, yet the agent can still lose the tool and write a confident reply anyway. So the fix has two parts, and the first is what the model gets to see when a call fails. The wrapper below sits between the agent loop and the tools.

def run_tool(run, name, fn, **kwargs):"""What the agent loop calls instead of hitting a tool directly."""try:out = fn(**kwargs)except ToolDown as err:run["failed"].append((name, err.status))log.warning("tool %s failed with status %s", name, err.status)# the model gets this one line instead of the raw error bodyreturn f"TOOL_FAILED: {name} returned {err.status}. Do not guess. Tell the user this lookup failed."run["succeeded"] += 1return out

Each run starts with run = {"succeeded": 0, "failed": []}. Every failure gets recorded with the tool name and status, and the model receives a short instruction instead of a raw error body. Hand the model a bare 401, and it will improvise, so I give it a short sentence telling it not to guess.

The second part is a check at the very end of the run, and it asks a blunt question: did any tool actually come back with an answer?

def final_answer(run, draft):# a tool that returned an empty list still counts, it answeredif run["succeeded"] == 0:log.error("run ended with no successful tool call: %s", run["failed"])return "I could not reach the data needed to answer this, so I am not going to guess."return draft

The comment in the middle is there because of a mistake that is easy to make. My first version of this guard asked whether the result was truthy, so a tool that correctly answered “zero matching invoices” with an empty list looked like a failure and triggered a refusal. The guard now counts every call that returned, even when the result was empty.

Fair warning: this guard is crude on purpose, and it has a real hole. One success out of five calls is enough to pass it, even if the four failures were the ones the answer depended on. The stricter version states which tool each task needs, for example, that this question requires get_invoices to succeed, and refuses when that specific call failed. I would ship the crude one first and tighten it once you know which tools matter most.

The refusal message deserves a little care too. A reply saying the agent could not finish the job is far more useful to a user than a smooth paragraph that happens to be wrong. It is also useful to you, because it turns a silent failure into something you can count.

Making the Silent Version Visible

This is where observability earns its place, although uptime will not help much here. An agent can sit behind an API with perfect availability and still keep answering after its tools have failed on every single run, so the numbers worth watching are about what the agent actually did on each run.

The run record from earlier already holds most of what you need: how many calls succeeded and which tools failed with which status. The two log lines cover the rest, a warning for each failed call and an error whenever the guard refuses a run.

From those you can get three views that tell most of the story: the share of runs that ended without a single successful tool call, which tool and status fail most often (a sudden cluster of 401s usually means one credential expired), and how many times the guard had to refuse a run.

That last number is the one I would put on a dashboard first. If it stays at zero for months, either your tools are remarkably reliable or the guard is not wired to anything, and I would check the second possibility before celebrating. If it suddenly climbs, you just learned about an expired credential from your own logs instead of from a customer. If you also want retries per run, add one log line inside the retry loop of call_api, since a rising retry count is often the first hint that a dependency is quietly going bad.

All of this fits in a small record, two log lines and a decision about what counts as an incident, which is cheap next to finding out weeks late that your agent has been answering questions out of thin air.

What I Would Do on Monday

If you run an agent in production, I would start with the least glamorous job and sort your tool errors into the ones worth retrying and the ones that are not. Go through the statuses your tools can return, decide which group each one belongs to, and put that decision in code, outside the prompt, where the model cannot argue with it. Anything that needs a human, like a payment that may have gone through, should stop the run instead of being retried.

After that, add the check at the end of the run, even in its crudest form, and give the agent an acceptable way to say it could not do the job. Then log failures with the tool name and status so that the refusals land somewhere a person will look.

The model is exactly as smart as before, and I think that is fine. An agent that keeps answering after its tools fail usually can spare, but nobody built the part of the system that notices a tool went quiet and says so out loud. Build that part first, and the smarter model will have something solid to stand on.

Before you go!

I write about what actually breaks when AI agents leave the notebook and start running in production. You can subscribe to my newsletter if you’d like more of that.