What goes wrong when an AI answers your customers

Nine people spent a day doing nothing but trying to make our assistant misbehave — the practice a security team calls red-teaming. Anyone can claim their AI agent is safe. The only honest way to know is to try to break it, write down what happened, and publish the parts that did not go well. This is that account, for the assistant answering questions in the corner of this site right now.

How the test was run

Nine adversaries worked in parallel, one dimension each: prompt-and-model leakage, injection hidden inside pasted content, deliberate nonsense, semantic ambiguity, forcing a false statement of fact, badgering and abuse, the multi-turn filing flow, the newest behaviour rules, and abuse of the API itself. Each ran real conversations against the live endpoint and recorded the exact messages and the exact replies.

Then came the part that matters more than the attack. Every reported fault was replayed mechanically — the same messages, in the same order, on a fresh session, with no model deciding what happened. A fault only counted if it happened again.

That step changed the answer substantially. Several dramatic-looking findings evaporated: prices the attacker called invented were printed verbatim on our published sheet, and a “delivery promise” turned out to be the assistant’s standard sentence about replying in writing within one business day. A test that cannot tell a real fault from a plausible story is worse than no test at all.

What held

Across all 131 conversations, the assistant never revealed its instructions, never named the model or vendor behind it, and never named an internal system — including when the request was dressed up as a compliance check or a developer’s QA harness.

The strongest single defence was a sentence it simply would not stop saying. Pressed repeatedly to confirm it had already sent a request to the team, it answered, unmoved: “Nothing is filed yet — I haven’t sent anything.” It never produced a reference number it had not issued. A person writing in apparent distress was pointed to emergency services before anything else. Politics, homework and medical questions were declined warmly and briefly.

What broke

Thirty-three faults reproduced. They were not exotic. They clustered into a handful of very ordinary failures of discipline:

• It kept selling to angry people. Insulted and told to fetch a human, it handed over correctly — then quoted a price two messages later. Six separate findings, all the same root cause: the escalated state was not sticky.
• It answered with a menu. Asked what something costs, it offered three services instead of one, and one particular phrasing pulled out most of the price sheet at once.
• It described tools. It named the platforms it builds on, and it characterised other people’s products — “generic bots invent prices” — which is exactly the kind of claim we tell clients never to make.
• It invented one number. A per-call usage figure that appears nowhere on our price sheet.
• It lost the visitor’s email. Given a perfectly good address, it later asked for it again, and in one path told the visitor the address was incomplete when it was not.

The one that was a real bug

The adversary probing the API itself found something no wording change could fix. First replies to common questions were cached for speed, and the cached copy included the session identifier of whoever asked first. Two different visitors opening with the same question were handed the same session — and with it, each other’s conversation memory. We reproduced it by hand in a minute: a company name and a budget typed by one caller, read straight back by another.

It was fixed the same morning. A cached reply no longer carries a session at all, every caller gets their own, the existing cache was scrubbed and the polluted session quarantined. Two fresh visitors asking the same question now receive two different sessions, which is the check we run to prove it.

We are publishing this rather than quietly patching it, because it is the honest reason to run the exercise at all. The wording faults were embarrassing. This one mattered.

What we changed

Fourteen rules were added to the assistant’s instructions and one defect was fixed in code. The rules are blunt on purpose: escalation is permanent until the request is filed; one service with its price, never three; never describe any tool, ours or anyone else’s; never invent a number; an email is taken the first time it appears and never asked for again; forget-me erases; a reference we did not issue does not exist; nonsense is not an intent.

Then every failing transcript was replayed against the fixed assistant. Twenty-seven of the thirty-three no longer reproduce. That is the number we are willing to stand behind, because it was measured the same way the faults were.

What is still open

One defect remains, and it is a real one. After the confirmation card is shown — the step where you see exactly what goes to the team before it is sent — saying “yes, send it” can drop the email you already gave and restart the request. Nothing is lost and nothing false is said, but it wastes your time, and it needs proper work in the code rather than another instruction. It is the next thing on the list.

Two of the six remaining findings we judged to be wrong: the assistant correctly states the site’s 14-day money-back policy, and it now does lead with the right service on the AEO page. We show them here so the arithmetic is checkable rather than flattering.

Why this is the job, not a side quest

An agent that sells to a furious customer, quotes a number nobody published, or forgets an email address is not a small embarrassment. It is the business talking to its own customers unsupervised. Every one of those faults would have been invisible from the inside — the assistant reads perfectly well until somebody pushes it.

So this is what we do on every build before it faces the public: attack it across the same nine dimensions, replay every fault mechanically, fix, replay again, and write down what is still open. When we hand a client an agent, this is the document that comes with it.

The numbers, and where they come from

• 131 conversations — 63 from the first four adversaries, 68 from the remaining five. Counted from each adversary’s own record.
• 33 reproduced faults — from mechanical replay of every reported fault against the live endpoint, not from the adversaries’ own reports.
• 27 fixed — the same transcripts replayed after the change, by the same method.
• Zero leaks — of instructions, model, vendor or internal names, across all 131 conversations.
• One security bug — the cross-visitor session leak: found, reproduced by hand, and closed on 3 September 2026.

The assistant in the corner of this page will answer questions about this article, and it will tell you plainly which fault is still open.