Field notes on AI coding agents: what they cost, where they break, and what shipped. Figures without a traceable source were cut.
Written with AI assistance. Figures without a traceable source were cut before publishing.
Klarna reported $4 million a year in savings and a 99.96 percent conversation engagement rate, the kind of pair of numbers that ends an internal debate before it starts. Six months later the same CEO said publicly that the company had overshot and was rehiring agents. The system had been fine on the repetitive work and then ran into complaints that needed empathy, recommendations that needed judgment, and escalations that needed a person on the other end of them. The savings were real. So was the rehiring.
The path that survives contact with reality starts much smaller than replacing 100 agents, with one person and one high-frequency task, before anything gets called an organizational capability. Here is what Klarna automated, which parts of it came back in-house, and the stages that separate the two.
The headline blared across tech blogs. But six months later, the same CEO admitted publicly: “We overshot. We’re rehiring agents.”
The promise of replacing human workers with chatbots sounds brilliant until you realize that, while AI excels at repetitive tasks, customer service requires nuance, trust, and the human touch, which are qualities that chatbots struggle to replicate.
The core problem lies in data complexity. Klarna’s AI system struggled with real-world scenarios—customer complaints that required emotional empathy, nuanced product recommendations, and handling escalations to human agents.
The lesson here is clear: AI isn’t a one-size-fits-all solution. While it can automate routine tasks, complex customer interactions still require human judgment. Klarna’s experience shows that enterprise AI adoption must start with small, high-frequency scenarios—not broad, generalized solutions.
The order matters more than the ambition. Start on one desk. Establish the real usage pattern there first, with one person and one task, and only then call it an organizational capability. Klarna ran that sequence backwards, and the $4 million showed up on the books months before the CSAT number showed up to argue with it.
Let’s start with the facts that made Klarna’s gamble seem reasonable:
Every one of those three numbers is real, and every one of them was collected before the hard cases arrived. That is the part worth sitting with. The metrics were not wrong. They were measuring the half of the job that a model does well, on a queue that had not yet been asked to carry an angry customer.
Six months after Klarna’s rollout, customer satisfaction scores fell to 68%. Why?
The AI couldn’t parse nuanced complaints about late deliveries or unclear policies, and it failed to escalate them properly. In 10% of cases it misclassified a complaint as a simple issue and left a customer waiting on a resolution that was never coming, and another 18% of the cases that did get escalated were mishandled after the handoff.
Humans pick up on tone, hesitation, and emotional cues, and the AI has no intuition for any of it. When a customer wrote “I’m really upset,” the AI answered with a generic apology instead of a solution. The 99.96% engagement rate says it could hold up the interaction volume; the 18% conversion rate says the emotional read was missing.
Trust runs on authenticity, and Klarna’s AI came across as robotic, so customers abandoned interactions early. Complaints about “cold responses” rose 10%, which is the number that told them a transactional bot was not going to carry a long-term relationship.
All three failures are the same failure. The system reads the words, not the situation. So it does the correct thing for a case it has misread. A CSAT of 68% is what that looks like from the outside: the queries still get answered, and the people asking them stop believing the answers.
The fix is not more model. It is deciding, in advance and in writing, which cases a person sees.
That is a routing question, not an empathy question, and it is answerable with the same numbers Klarna already had. The 10% misclassified as simple and the 18% mishandled after escalation are both routing errors with a paper trail. You can build a rule against a number like that. You cannot build one against “be more human.”
Instead of full automation, companies should follow this evolutionary path:
The first stage is manual process optimization: map every customer service workflow before any of it goes near a model. Refund processing at Klarna ran 17 steps, and they cut it to 5 by hand.
The second stage puts AI on top of those cleaned-up workflows as a helper, not a replacement.
The third stage is where the routing gets built. Klarna’s hybrid model steered customers to human agents automatically when the AI detected emotional distress or a policy exception, and that, paired with clear handoff protocols, lifted first-contact resolution by 22%. I don’t think this transition fully scales—without human oversight, the system struggles with edge cases, and the quality drops.
Only in the fourth stage does the automation run wide, with human review as a spot check rather than a gate. Klarna’s AI now handles the simple issues and a human agent audits 10% of escalated cases weekly, which holds quality while cutting costs by 60%.
Getting from the third stage to the fourth is the part that needs care. Klarna found that going straight to full automation degraded service quality, because the system couldn’t handle edge cases without a person in the loop.
What the ladder bought Klarna was the 60% cost reduction without the service collapse, and the reason it held is that each rung was allowed to fail cheaply before the next one got built.
For solo founders building AI customer tools, start small and focus on these:
Define the task narrowly instead of building a general-purpose chatbot, and target one specific workflow. Shopify store owners needed help extracting product reviews, and building for exactly that is what kept the thing from being overengineered.
Build the “human override” before you build anything clever. Spotting a frustrated customer early and routing them to a person is what keeps escalation time down, and it is the one feature that pays for itself on the first angry ticket.
Track the empathy number next to the throughput number, which means watching CSAT scores alongside task completion rates. One A/B test cuts against the intuition here: the blunter, more pressuring script was the one that lifted lead conversion from 10% to 18%, not the gentler one. That is the argument for watching both numbers at once — optimize on conversion alone and you ship a bot nobody wants to talk to twice.
I could be wrong here. This is one company, one quarter, and Klarna had reasons to publish both the $4 million and the retraction that have nothing to do with what works for a team of one. But the shape of it keeps showing up: the number that gets measured first is the number the queue was already good at, and the number that ends the experiment arrives two quarters later wearing a customer’s name on it. If you are going to instrument one thing before launch, instrument the handoff — how often the bot decided a case was simple, and how often a person later disagreed. Klarna’s 10% and 18% are exactly those two counters, and they are the only numbers in this story that would have predicted the rehiring before it happened.
Also readable on Telegraph.
Read next
Part of ai-coding-field-notes — field notes on AI coding agents. Found something wrong, or shipped something similar? Open an issue — corrections are the point.