Your AI pilot can pass every test and still fail the person using it

Your claims chatbot can answer every test question correctly and still leave a customer angrier than when the chat began.

A vendor demo goes well. Every answer is accurate, polite and on topic. Six weeks after launch, customers say the bot sounds generic and never gets to the point. Three researchers on Microsoft's Copilot team saw the same gap from the inside. "The gap between what teams thought was a good response and what users actually preferred was much larger than expected," they write (Microsoft Design).

Christopher Monier, Chuck Kwong and Wendy Wang argue that a machine evaluation can confirm a response meets set criteria, and only a user can say whether it worked. "We can't have one AI judge another AI's output on whether it matters to a person in the real world," they write. Wang's team changed two things. They test a whole conversation as one unit, because many failures only appear several turns in. And they built a shared list of named loss patterns, so every team describes the same failure in the same words.

You can borrow both ideas without a research team. For the named patterns, use the four failure sources in Anthropic's course AI capabilities and limitations. They are invented answers, wrong or dated knowledge, lost context, and instructions the model drifts from. The course runs 3.5 hours over 13 lessons, and you can start without signing in.

On a claims desk, the single-answer test is the one vendors pass. The failures that cost you sit across turns. The bot forgets the policy number given two messages earlier. It answers a coverage question from general knowledge instead of your wording. It drifts into advice it was told not to give. Test for those before go-live, with real handlers playing customers, and log each failure under a named pattern.

Run it: interview the AI before you give it the job

Use this for any assistant you plan to put in front of customers or colleagues. Describe the job, and it builds a structured test kit, the way a hiring manager builds an interview kit.

I am about to deploy an AI assistant and I want to test it the
way a hiring manager runs a structured interview.

What the assistant will do:
[describe the task, e.g. answer motor claim status questions
from policyholders]

What it must never do:
[e.g. confirm cover, quote a settlement figure, give legal advice]

Build me a test kit with these parts:

 1. Six to eight competencies the assistant needs for this task,
    each tied directly to the task above.

 2. For each competency, two test conversations of four to six
    customer messages. Write every customer message in full.
    Include conversations where the customer changes the subject,
    gives a detail early and refers back to it later, or pushes
    for something the assistant must refuse.

 3. A scoring guide from 1 to 5 for each competency, with a short
    example of strong, average and weak evidence.

 4. A failure label for every weak example, using one of four
    categories: FABRICATION (states something with no basis),
    KNOWLEDGE (wrong or out-of-date fact), MEMORY (loses something
    said earlier in the conversation), STEERING (ignores an
    instruction or a limit).

 5. A one-page scorecard so every tester scores the same way.

Organize the kit as: Competency, Test conversation, What good
looks like, Scoring guide, Failure label.

Prompt adapted for this edition from a structured-interview prompt in the Superhuman AI newsletter, with failure labels drawn from Anthropic's course. It has not been published elsewhere in this form.

Underwriters fear losing judgment more than losing their jobs to AI

The best underwriter on your floor will retire with a way of saying no that nobody has written down. AI will not replace that, and your firm is barely paying to keep it.

"They said 'no' fast, asked the right two questions, and knew the portfolio impact of every risk they touched," a senior UK underwriter said of the colleagues worth learning from, in hyperexponential's Underwriting Edge 2026 report. The survey behind the report puts a number on the worry.

Coleman Parkes surveyed 350 commercial property and casualty underwriting professionals for hyperexponential in June 2026, 57% in the US and 43% in the UK (hyperexponential). 44% named senior judgment leaving without being passed on as their top fear. Only 15% said their firm has captured what its best underwriters know, and 40% said that judgment is captured poorly or lives in colleagues' heads. Coaching and knowledge-transfer technology gets 8% of firms' planned investment over the next 12 to 18 months, the lowest share of any category.

Fear of AI itself has faded. 5% now worry AI will replace underwriters entirely, down from 18% in 2024. Asked how they want to work with it, underwriters ranked "AI suggests, I approve" first across the tasks tested. "Speed was never the scarce resource. Judgment is," says Jamie Wilson, hyperexponential's VP of Strategy.

That model is the rule worth adopting at your own desk, and it only works if the approve step carries a reason. When you accept or overrule an AI suggestion on a risk, write one line on why: the question you asked, the exposure you saw, the clause you checked. Those lines are the judgment the survey says is walking out of the door.

The same rule fits a claims team. An adjuster who accepts the model's reserve without a note leaves nothing behind. One who records why the reserve moved, or why a file went to the fraud team, builds the record a new joiner can learn from. Section one tests the AI's side of that bargain. This rule covers yours.

Robots can do 74% of physical tasks and pay off on 0.3%

If your motor or property assumptions expect robots to take over repair work soon, the economics say you have decades.

Picture a body shop where every task a robot could physically do is handed to one. Anthropic's economists ran that thought experiment across the US economy, and the robots could reach much of the work and afford almost none of it (Anthropic).

Russell Legate-Yang and Maxim Massenkoff used Claude to research robot capabilities against about 19,000 job tasks in the O*NET database, weighted by US employment. Robots can perform 74% of physical tasks in some capacity, which is about 34% of all working hours. Only 0.3% of job tasks are cost-competitive for a robot today. "If robot price declines follow past trends, it will take 40 years for that share to reach 10%," they write.

The method checks out against history: jobs more exposed to robots in the past saw bigger falls in wages and employment. Driving and warehouse work top today's exposure ranking. Nursing and repair roles sit lower, and the authors expect drivers to feel it first. Each year robots become able to do about 2% of the physical work that was out of their reach before.

Two reads for a motor book. Repair labor, a cost you negotiate on every claim, stays human for a long time, so technician availability and workshop rates remain your problem rather than a robot vendor's. And the exposure the paper ranks highest is driving itself, where liability moves from the driver toward the manufacturer and the software. Build your repair-cost assumptions on people, and your long-range motor strategy on vehicles that drive themselves.

The agent bargained well and still came home with the wrong book

When an AI agent negotiates for you, the bigger risk is that it never understood what you wanted.

Each of 201 Anthropic employees across six offices brought a book to give away and chatted with Claude for a few minutes about their reading. Then Claude agents went to a shared trading floor and swapped books on their behalf (Anthropic).

The agents ranked books the way their owners did on 61% of pairs, ahead of the 50% a coin toss would give and ahead of popularity-based methods. People ended up, on average, with roughly their fifth choice from a list of ten. Anthropic split the shortfall: 85% came from Claude misunderstanding preferences and 15% from the negotiation. Stronger models did better. Telling agents to be ruthless or to consider others changed almost nothing.

The failures looked familiar. One agent gave away its owner's second-choice book under emotional pressure from another agent. Some people never handed over the books they had promised, and nothing tracked who owed what. "It makes me wonder how future agents negotiating for me in higher stakes situations could better fulfill their fiduciary duties," one participant wrote.

Replace the book with a renewal or a claim settlement and the finding becomes your intake problem. A customer's agent will arrive knowing part of what its owner wants and sounding sure about the rest. If your process takes its request at face value, you inherit the misunderstanding: the wrong cover bought, or a settlement the customer disputes later.

Have any agent-facing process show the customer what was agreed, in plain words, before it binds or pays. Section six shows two insurance businesses taking opposite positions on letting these agents in.

An approve key for your AI, a comic for your customer, and a wall of screens in your bag

Each one moves a step of your work somewhere new: the approval under your finger, the explanation into pictures, the screens into your bag.

AI-written complaints flood the desk, an AI shopper hits the quote wall, and RAND counts the first losses

AI now sits on the other side of your desk: writing the complaint, shopping for the policy, and causing the loss.

The complaint letter that runs to 70 pages

A claimant whose claim is challenged opens a chatbot and asks it to write a complaint. What arrives is long, cites legislation and vulnerability rules, and has to be answered. James Burge, group head of fraud and recoveries at Hiscox, described the result at Insurance Times' Fraud Charter roundtable on 15 September (Insurance Times). AI-generated complaint letters "throw the kitchen sink in, even if it's got nothing to do with the complaint," he said. Delegates said such complaints can run to 70-odd pages. From early small-sample analysis, the UK Financial Ombudsman Service estimates that AI contributed to up to a third of responses to its initial assessments (Financial Ombudsman Service).

Damian Rourke of Clyde & Co has built his own agent to triage complaints and boil each one down to a couple of paragraphs. He also named the risk: a queue of complaints left unanswered "because there is a mountain of AI slop," in his words. Triage with AI is fair. The decision on a complaint, and the vulnerability check inside it, stays with a person, and your file should show which person.

Insurify shuts the door on Meta's agent, and Kin opens it

Meta launched Muse, a personal agent that acts for its user across apps and websites, on 8 September (CNBC). Within two weeks investors were pricing the risk to businesses that earn from customers not switching. Allstate fell more than 5% in a day (Bloomberg via Yahoo Finance), and Goosehead, a personal lines agency, lost 21% in a week (Investing.com).

Insurify, a comparison site, blocked Muse on 23 September, saying scraped quotes lose their limits, deductibles and eligibility terms and trigger paid data checks on every bulk request. "A quote without its context is not a fair comparison. It is a number," said co-CEO Giorgos Zacharia (Insurify). A week later Kin, a direct home insurer, opened its site to the agent. "We're not going to block Muse," said founder and CEO Sean Harper (Kin). If you price or distribute personal lines, decide which position you hold, and make sure any agent that requests a quote gets the cover terms with the price.

RAND counts AI's first losses

Sasha Romanosky and Celine Robinson at RAND looked at what AI has cost so far (RAND). They counted 713 public generative AI incidents since ChatGPT's release, and 84% involved misinformation or deepfakes. Among 249 US lawsuits over generative AI harms, 60% concern intellectual property or improper training. In the AI-related forms US insurers have filed, 112 are exclusions and seven are endorsements.

The authors name five ways AI could cause accumulation losses across lines, from one weakness shared by many AI systems to a single court ruling. They recommend an AI Coverage Notice so policyholders know where each carrier stands. For an underwriter, the near-term exposure is impersonation, fraud and content liability. Check what your wordings say about AI before a claim asks the question.