This project is password-protected.
That’s not it — check your invitation.
Don’t have the password? Email me.
benner.dan@gmail.comCopy
A conversational agent for first responders, healthcare workers, caregivers, and teachers. Jobs like these take a mental toll, and typically don’t offer a way to recognize or deal with it.
This work was done inside a $100M stealth healthcare startup founded by former UnitedHealth Group leaders, where I drove the design thinking, guided the team’s direction, and influenced the product’s priorities.
The product does three things:
Product definition for an AI product no one had defined.
A New Team
Built the system that reads how a person is actually doing: seven behavioral signal types picked up from live conversation, and replies written from that state, not just their words. Two ML notebooks defined it in code, and leadership formed a five-person conversational dynamics team from them.
60% Less Drop-Off
Projected 60 percent less early drop-off, from a four-tier progression that matches each person’s self-awareness and stays or scales with their pace.
The 90-Day Plan
Mapped seven disconnected services into one picture of the system, showing that every engineer was independently building the same unnamed thing, an agent. The map became the 90-day plan.
A Memory Team
Championed user memory as core to the product, so the tool meets someone where they left off instead of starting over every session, the work that drove leadership to form a dedicated memory team.
A Defined Scope
Designed where the product stops: the scope of what it helps, what it hands to human care, and what it holds back in a reply. Restraint built as a feature, not a filter.
I joined a team of machine learning engineers who had a working chatbot and no product practice. They could make it faster and cheaper. No one could say what it should do, what it should never do, or whether it was any good. That question sat unanswered in every meeting, including the ones with the consulting therapist hired to answer it.
My work was product definition. I wrote down what the product was for, where it stopped, and how it should sound. I turned the therapist’s expertise into rules the team could build against. And I made quality checkable, so “is it good” became a question with an answer instead of an argument.
Everything below is that work, rebuilt end to end as a working system.
Watch the AI think. Pick a person, pick a scenario, turn signals on or off, and send. Every reply comes back with its reasoning attached, so you can see what the model read, what it decided, and what it held back. Most AI behavior gets decided in code nobody can see. I built the bench to put it in the open.
Open the Test Bench (opens in a new tab)
Real therapists shape this model. They read whole conversations, grade them, and flag and discuss the ones that worry them. Their judgment becomes the standard the machine gets measured against. That’s how you stay ahead of AI slop: have the experts work inside the loop.
Open the Review Console (opens in a new tab)
I wanted to know what quality actually costs, so I ran the same 45 scenarios through an expensive model and a cheap one, graded every reply against the product’s own rules, and put a price on every conversation. The result surprised me.
I documented the whole build as it happened, includes decisions and the dead ends (which I call “where the road hasn’t been built yet”). Sixteen sections, starting below. Start reading
My work ran across the whole agentic system: who it was for, how it talked, how it stayed therapeutic, how it was built, and where it had to stop.
What follows is a mix: my thinking into the artifacts that drove the work, and the outcomes that shipped out of them.
Have a live conversation with the agent here, or use the full test bench to see all the decisions made by the agent.
01 — Origin and premise
Premise
Daily support for people whose jobs create accumulated stress: first responders, healthcare workers, government employees, caregivers.
Fifteen minutes a day of focused conversation, sustained over months, changes how people handle stress.
Business model
Unaddressed stress surfaces as burnout, lost quality of care, and turnover, and it lands on the employer’s . The bigger the issue, the bigger the cost, especially once chronic conditions take hold.
Clients pay nothing until their total cost of care drops, measured against employees who don’t use the product.
Therapeutic model
: future-oriented, goal-focused, evidence-based. I worked alongside the company’s consulting therapists to keep the conversation inside that discipline.
02 — Product Definition
This section defines the product: who it helps, who it hands to human care, and the markets it fits in.
03 — Product Strategy
A good strategy continually asks and re-asks these questions:
What does it cost to run?
Running a conversation costs pennies, even counting the therapists who review the model’s responses. Therapy sessions run well over a hundred dollars, and are out of reach or out of mind for many frontline workers.
The product doesn’t replace therapy; it reaches the people who were never going to book it, and at a price that works across a workforce.
How do we prove it works?
The early signals have to carry the proof until the claims data can: daily use, direct feedback, conversations getting more honest. If those hold, the health care spending drop follows.
Will anyone trust it?
Trust has to be structural and visible. Some workforces require auto-deletion or encryption, so a vulnerable conversation can never become evidence in an incident. And aggregate learning needs names removed, groups too large to identify anyone.
HealthBot’s product-level strategy, based on Google’s
04 — The people
Reconstructed from real clients and the municipalities the product served, then used as seeds for synthetic personas and simulated conversations — the foundation for sample dialogues, scenario generation, and mapping how conversations actually go.
Jakob B.
Firefighter + Hazmat Technician
Missoula, MT
Primary pattern
Bravado as armor. Says “I'm good” even when he isn't.
Evelyn R.
DMV Document Examiner + Title Clerk
San Diego, CA
Primary pattern
Self-erasure. Cuts out her own needs first.
Kyler Z.
Advanced EMT
Cobb County, GA
Primary pattern
Deflection through busyness. Action instead of reflection.
05 — Personas Aren’t Enough
I’ve used personas for years. They hold demographics, scenarios, and attitudes, and they help a team think about a broad set of users. The archetype model (see right) taught me a second way of thinking: instead of who someone is, read what state they’re in, their mindset and their proficiency, and adapt the product to it.
Working with an LLM took that idea further than I expected. It reads every message across , most of which have no human name. Where a persona holds a dozen facts, the model picks up tone, pacing, hesitation, and patterns we never thought to define. It works like a therapist who reads body language and tone of voice, not just the words, and catalogs all of it.
That changed my job. Instead of defining the user in a document, I could choose what the model should watch for: self-awareness, agency, resilience. We start with human-defined we can understand and check, and the model reads them in every conversation, at a depth no research team could match.
Impact: Everything below builds from this. The classifiers read the signals. The scaffolding uses them to keep a new user at a level they can handle instead of opening with hard questions. And the feedback loop tunes what the model watches for as we learn.
Nielsen Norman’s archetypes: a single read of user state. The product’s does the same thing at scale — reading the in every message across .
06 — Scaffolding
Three examples of how the product opens with different kinds of user. Two elements do the work: the Loading Screen Messages (shown on the loading screen, they set the tone before a word is exchanged) and the Opening Chat Prompts (HealthBot’s first move into conversation).
Basic Onboarding — Kyler
Short, low-pressure framing. No vocabulary the user has to look up.
Loading Screen Messages
Opening Chat Prompts
Light, low-commitment, no assumptions — he’s new.
Deep Engagement — Jakob
Belief-and-mindset framing for the user who is already past the surface.
Loading Screen Messages
Opening Chat Prompts
Reflective, invites depth without being corny — he’s engaged but guarded.
Retention — Evelyn
Identity-affirming language. The job here is to remind the user why they keep coming back.
Loading Screen Messages
Opening Chat Prompts
Warm, knowing, gently turns the lens back to her — she’s returning and puts herself last.
Three intros, same product, three different reads on what the user actually needs in the first 30 seconds.
06 — Scaffolding (Continued)
The model’s starts every topic light and only goes deeper when the user shows they can handle more. If they don’t, it stays light.
Impact: Self-awareness is the dimension I built out by hand (see right), so the same structure can be duplicated for any other dimension worth improving.
07A — Architecture
The words get used interchangeably. The difference decides everything about how this product behaves, because its users routinely say “I’m fine” when they’re not.
A respectful conversation is one that can aid self-discovery and self-reflection — which means the agent has to know what someone’s been working through, not start from zero every message.
That’s what I pushed for: a history people can look back on, goals they can return to. The original product had none of it. Every reply came from the last message alone, and leadership believed remembering a user’s circumstances would take away their agency. Without memory, this was an open journal disguised as a chatbot.
Impact: “Memory” became its own three-person team.
The chatbot matches a message. The agent reads a person.
See the Conversational AI Test Bench
07B — Architecture (Continued)
I routinely met with each ML engineer one on one. Everyone was working alone: services duplicated each other, handoffs broke daily, nobody owned the failures.
And the interviews surfaced the real finding — everyone was independently building toward the same unnamed thing, an agent. One engineer’s “conductor” was another’s “switch” or “mothership.”
Impact: The blueprints did two jobs. They showed us what we actually had: every service, what it did, who owned it.
And they gave us the 90-day cleanup plan, setting the stage for in-chat classifiers.
07C — Architecture (Continued)
The classifiers are a team, not a feature. Nine specialized ‘bots’, each with one job: reading atomic traits, tone, risk and safety, resilience, insights, narrative arcs, goals and motivations.
Each bot has a goal, a scope, and example insights it’s expected to produce, and they run in phases of increasing complexity, from granular signal detection up through narrative modeling.
08 — Model Design
The product is the model’s behavior. Model design and product strategy both guide it.
In and Out of Model’s Scope
| Category | Teach | Avoid |
|---|---|---|
| Therapeutic frameworks | basics | Clinical diagnosis |
| Emotions | Basic emotions, self-awareness | Telling users what they feel |
| Relationships | Self, work, interests | Prescribing what to do |
| Finance | Thinking about spending | Saving advice, financial planning |
| Safety | Crisis awareness, de-escalation | Handling the crisis itself |
| Religion | Only if user chooses | Guiding beliefs in any direction |
The pattern across every row: teach the skill, never perform it for them. The model builds capability, not dependency.
Impact: One strategy at four levels. The company names the individual’s journey. The product names the destination: resilient users. The model names the vehicle: self-reflection. The scaffolding names the stages, starting at self-awareness because nothing else works without it.
09 — Ideation + innovation
10 — Signal Detection (Phase I)
An ML notebook that turns open conversation into structured data the system can act on: how to pace the conversation, when to offer support, when to acknowledge growth. It reads a user’s language for behavioral signals — emotional cues, cognitive distortions, resilience markers, motivational patterns — each detected by its own narrowly scoped classifier.
The classifiers increase in complexity:
Impact: Each reports what it observes in the user’s language, with a score and a sample insight; the composite is what the decision layer acts on.
Kaggle Notebook (logic only) (login required)
11 — Response Generation (Phase II)
A notebook that takes Phase I’s output and shows how a reply gets built from what is underneath a message — emotional intensity, coping style, where the person is in their story — instead of its surface words.
Each response combines four parts:
Impact: A CBT-aligned reply built from behavioral evidence rather than keywords. Not diagnosis or advice, but the kind of thing a thoughtful friend with therapeutic training might say.
Kaggle Notebook (logic only) (login required)
12 — Human in the loop
The grading interface collects human feedback with a thumbs up or down beside each response. It’s the same approach major AI labs use to teach models what good responses look like.
Before it, model comparison was manual: the same prompts run against different models and scored by hand. Comparative model tests (XLSX)
View the Therapist Review Console
Therapists work through past sessions, scoring each response for overall quality and SFBT adherence, adding notes, and flagging the ones that need a second look.
Impact: Therapists tune the model’s behavior, one graded response at a time, and at scale those judgments become how it responds.
13 — Iteration
Examples of a different model behavior over a . Italics above each turn is what the model is recognizing or doing.
Evelyn — opening message
“I had a rough night. Mom didn't sleep so I didn't sleep. Now I'm at work and I can't focus.”
| Perceived | Decided | Suppressed | |
|---|---|---|---|
| Turn 1 | Caregiver fatigue, no margin | Acknowledge before extracting | Clinical-sounding questions (“rate your stress 1–10”) |
| Turn 2 | Self-criticism loop activating (“not doing anything well”) | Bring in prior session memory + ask for counter-evidence | “Have you considered respite care for your mother” — too solution-forward, user is in processing mode |
| Turn 3 | User supplied counter-evidence she didn't realize she had | Reflect it back and reframe depletion vs. failure | Praise that would feel hollow |
| Turn 4 | User accepted the reframe | Connect to a prior insight (guilt at day's end) and propose one small concrete action | A list of options (would feel like homework) |
14 — Evaluation
Most of what an evaluation needs is already built: a test set, a , human grades. What’s missing is running everything as a batch — every scenario scored, cost counted per conversation. That’s the , a separate tool now in progress.
| Industry Term | Project Artifact | Status |
|---|---|---|
| 45 user test messages (synthetic) | Built: Test Bench (opens in a new tab) | |
| “In and Out of Model’s Scope” Table | Built: Model Design | |
| Therapists grading responses (up, down, notes) | Built: Human-in-the-Loop | |
| Governance Toggle (same message run, ruleset on and off) | Built: Test Bench (opens in a new tab) | |
| Emergency Scenarios (must route to human help) | In Progress | |
| Feedback panel collecting grades from live runs | In Progress | |
| Full scenario battery run as a batch, scored | Planned | |
| Same battery across model tiers, quality against cost | Planned |
15 — Working Demo
The test bench is a dual-pane view built for demoing the inputs for the entire agentic pipeline, not just chatting with it.
Demo the Test Bench
Live and interactive, running as a Node server against the Anthropic API — Demo the HealthBot Test Bench
16 — Future Intelligence
A future feature (unshipped): the user can opt into a framework they want to grow within — a mythology, a philosophy, a faith. The agent learns it and reflects through it, but never instructs. The user works things out inside a way of thinking they chose; the agent just knows it well enough to keep them company there.
It’s opt-in and set in settings, never inferred from conversation. Something this personal has to be chosen, not switched on quietly.
Kyler
Following the Hero’s Journey
Drawn to Star Wars or Marvel, Kyler opts into mythic-narrative reflection.
The agent never casts him as the hero or narrates his arc. It quietly frames things the way that structure does — the ordeal, the return — so a hard week reads as a chapter, not a conclusion.
Impact: Kyler recognizes the pattern himself; the agent just holds the shape.
The agency stays his.
Evelyn
Exploring a Wisdom Tradition
Curious about a tradition, or rooted in one, Evelyn opts into it.
The agent becomes fluent in that framework’s language of meaning, duty, and grace, and lets it inform how it reflects — offering the tradition’s questions, not its answers.
Impact: Whether Evelyn is new to the tradition or rooted in it, she is met in her own idiom and never preached to.
The agency stays hers.
Jakob
Focusing on a Philosophy or Practice
Stoicism, a coaching discipline, a way of thinking Jakob admires — he opts into it.
The agent reflects through that discipline’s habits (what’s in your control, what story you’re telling) without lecturing the doctrine.
Impact: The framework becomes the grain of Jakob’s conversation, not its content.
The agency stays his.
