AI autonomy, safety, and the responsibility of serving users who need it most
When building AI systems for vulnerable populations, every design decision carries weight. Too much control and you lose the naturalness that makes conversations helpful. Too little and you risk responses that could genuinely harm.
We’ve spent years building conversational systems for charities & health services. Our work sits at the uneasy intersection between innovation and duty of care. To understand how we navigate that tension, we sat down with Jamie our CTO, who’s led Japeto’s approach to responsible AI from the start.
When Vulnerability Meets Innovation
There’s one vital piece of feedback that has stayed with our team. A neurodivergent user told us that accessing advice through our Pat chatbot was transformative, not despite it being automated, but because of it. Speaking to a human about sexual health had felt impossible for them. A chatbot offered something no person could: a judgment-free space where questions could be asked at their own pace, without fear or social anxiety.
This wasn’t proof that AI is better than people. It was proof that different routes to help matter. For some users, automation isn’t a downgrade, it’s access. The chatbot didn’t replace human services; it filled a gap that would otherwise have left someone without support at all.
Knowing When Not to Use an LLM
Today when building chatbots is discussed, there’s one architectural question that pops up first: Do we use an LLM? For many, the answer is likely “yes, but we’ll constrain it carefully.” They debate how much control to give the LLM, assuming its use is inevitable.
But for all their power, LLMs can still be a bit unpredictable. They can fabricate information, misinterpret nuance, and produce inconsistent answers to identical questions. For a user asking about their housing rights, benefits eligibility, or health options, a 90 percent accurate answer isn’t acceptable.
Imagine a housing charity chatbot receiving this message:
“My landlord is threatening to evict me because I complained about the mould. I’ve got two young kids and nowhere to go.”
An LLM might try to respond empathetically and sound as human as possible, but users can usually spot when a chatbot is faking feeling, and many don’t want synthetic condolences in the first place. They come for information, not emotional performance. But what if it hallucinates a legal deadline that doesn’t exist? What if it misunderstands tenant protections or omits crucial advice about emergency accommodation? A more scripted rule-based system, by contrast, might feel more rigid, but every line it produces comes from verified sources.
We haven’t seen our LLM-powered systems backfire, largely because we’re deliberately cautious. In most cases LLM responses are turned off for our chatbots involving vulnerable people. We use scripted, carefully reviewed responses for the highest-stakes conversations. AI can assist by helping draft or refine these scripts, but it never replaces you as the author. Only once the foundations are solid do we consider introducing any degree of AI autonomy.
The technology is there, but restraint is its own kind of innovation.
Restraint as a Design Principle
In most of Japeto Chat’s systems, LLM functionality starts switched off. Early deployments rely entirely on scripted, reviewed responses. This isn’t caution for its own sake, it’s how trust is built. Stakeholders can see that the chatbot works safely before any generative components are introduced. As confidence grows, limited autonomy is added: not to replace the rules, but to complement them.
“By the time you turn on LLM responses, you can use past conversations to make sure that you have created scripted responses to all the critical questions, and ensure that you have given the LLM enough information to answer a wide range of queries with confidence.”
This gradual introduction of autonomy overturns scepticism. Instead of asking “Will this work?”, our partners start asking “How do we expand what’s already working?” a much healthier question.
Oversight Without Obstruction
As autonomy increases, so must oversight, but not always in the ways you’d expect.
Human-in-the-loop approval (where someone reviews responses before they reach users) sounds ideal but often isn’t practical for live chatbots. Users expect immediate responses. Making them wait for human approval would destroy the experience that makes chatbots valuable in the first place.
Instead, we focus on “human-on-the-loop” monitoring: regular human review of past conversations to verify the LLM is behaving as intended. This isn’t a nice-to-have; it’s vital. You need eyes on the actual conversations happening in the wild.
Automatic safeguards form the first line of defence, keeping bots on topic and away from dangerous territory. But algorithms alone aren’t enough. At scale, smart monitoring means implementing systems that flag conversations needing extra review. Japeto’s health scoring and safeguarding features do exactly this: they help human reviewers focus their attention where it matters most.
Why Predictability Still Wins
The temptation to rely on LLMs is understandable. They sound natural, scale effortlessly, and can simulate empathy in ways that rigid systems struggle to match. But sounding natural isn’t the same as being safe.
Take food assistance support as an example. A parent messages a charity chatbot:
“I just lost my job and I’m scared I won’t be able to feed my kids.”
An LLM might respond with genuine-sounding care, it might even suggest programmes or services that appear helpful. But what happens if one of those programmes doesn’t exist anymore? Or if eligibility criteria changed last month?
Japeto’s rule-based approach handles this differently. The system recognises a food-assistance query, uses structured follow-up questions to gather details, and presents verified options from an expert-maintained database. Every message is reviewed and pre-approved. The tone may be slightly more formal, but the information is accurate and the advice traceable.
Technical Optimism, Practical Necessity
Technical Optimism, Practical Necessity, or Both?
Are we moving toward more autonomous LLMs because the technology excites us, or simply because it’s convenient?
Honestly? Both. And that’s not necessarily a problem.
In high-stakes contexts, autonomous AI should never replace humans entirely. That’s a line we need to hold firm. But there’s an enormous difference between replacement and augmentation.
Charities and public services face a constant challenge: unlimited need, limited resources. They always need to do more with less. This is where thoughtfully deployed AI becomes genuinely good for users, not because it’s clever, but because it enables 24/7 support that would otherwise be impossible. It answers the straightforward questions so human experts can focus on complex cases. It provides that judgment-free space for users who need it.
When Autonomy Makes Sense
There are contexts where the trade-offs of LLMs make sense, internal tools, low-stakes informational chatbots, creative applications, or environments with strong oversight and well-resourced monitoring. In those cases, the benefits can outweigh the risks.
But for systems that guide people through housing crises, benefits claims, or healthcare decisions, the equation changes. Vulnerable users need certainty. They deserve systems designed for their protection, not experiments in technological potential.
The Question That Really Matters
The conversation around AI ethics often focuses on technological sophistication, which model, what temperature settings, how to fine-tune. But the real question is simpler:
Does this system serve the people who need it?
Sometimes that means giving the LLM room to handle the simple stuff so humans can focus on what matters most. More often, it means knowing when to hold back.
Conclusion
Innovation in AI will continue to accelerate, and LLMs will get safer, smarter, and more controllable. But no model update will ever remove the obligation to think critically about context. In high-stakes, emotionally charged environments, the cost of a single wrong answer can far outweigh the benefits of automation.
At Japeto, restraint isn’t a limitation. It’s a design choice, one that ensures the technology we build genuinely helps the people it’s meant to serve.
