It's 4:50 p.m. on a Friday. A patient calls and says her chest feels "tight, but probably just stress." Your front desk is closing out the day. The nurse who would normally take that call is in a room. The call goes to voicemail, and the patient decides to wait until Monday.

Every practice has a version of this story. It's the reason AI triage agents are one of the fastest-moving categories in healthcare automation — and it's also the reason they make clinical leadership nervous. The same system that catches that call at 4:50 p.m. is a system that is, in some sense, making a judgment about a symptom.

So let's be precise about what an AI triage agent should and shouldn't do, and what guardrails belong around it before it ever speaks to a patient.

Triage is routing, not diagnosis

This is the distinction that makes everything else work.

A triage agent's job is not to determine what's wrong with a patient. Its job is to determine how fast that patient needs to be seen, and by whom. Those are different problems with very different risk profiles. "This person describes crushing chest pain and shortness of breath, route to emergency care now" is a routing decision. "This person is having a myocardial infarction" is a diagnosis. Your agent should be doing the first and never attempting the second.

This isn't a semantic point — it changes how you configure the system. A routing agent is measured on whether patients landed at the right level of care in the right timeframe. It's allowed to be conservative. Over-escalating a sore throat to a same-day slot costs you a slot. Under-escalating chest pain costs considerably more. When you build the agent around routing, "when in doubt, escalate" becomes a coherent design principle rather than a failure.

Guardrail 1: Start from your protocols, not the model's judgment

The most important question to ask any triage AI vendor is simple: whose clinical logic is this?

Telephone triage is not new territory. The Schmitt-Thompson protocols — roughly 1,000 pediatric and adult telephone triage guidelines maintained by Dr. Barton Schmitt and Dr. David Thompson, and updated annually against current medical literature and clinical review — have been the backbone of nurse triage lines for decades. Peer-reviewed evaluations of them cover millions of triaged encounters. Whatever your practice uses, the point holds: there is an existing, reviewed body of triage logic, and it took a long time to build.

An AI agent that improvises its own clinical reasoning is throwing that away. An AI agent that executes an established protocol set — or your practice's own written protocols, if you have them — is doing something much narrower and much safer: conducting the interview consistently, every time, and applying the disposition rules you already agreed to.

Ask the vendor to show you the actual decision logic. If they can't show it to you, your nurse manager can't review it, and if your nurse manager can't review it, you can't stand behind it.

Guardrail 2: Define the escalation ladder before you go live

Every triage agent needs an explicit, written ladder of what happens at each urgency level. Not "the agent flags urgent cases" — a specific ladder, with owners and time bounds. Something like:

• Emergent — agent instructs the patient to call 911 or go to the ED, stays on the line to confirm understanding, and pages the on-call clinician immediately.
• Urgent — agent books into a same-day or next-morning slot and notifies the clinical inbox within minutes.
• Routine — agent schedules normally and writes the intake summary back to the EHR.
• Out of scope — agent stops, says so plainly, and transfers to a human.

That last rung is the one practices skip, and it's the one that matters most. Your agent needs a well-defined "I shouldn't be handling this" state: the patient is distressed, the answers are contradictory, the symptom doesn't map to any protocol, or the patient simply asks for a person. The measure of a good triage deployment isn't how few calls it hands off — it's whether the handoffs happen at the right moments.

Guardrail 3: Build the safety net for the calls that aren't about symptoms

Some of the highest-stakes moments in patient communication never present as a clinical complaint. A patient calling to reschedule may disclose something about self-harm. A medication question may reveal a substance use problem. A routine follow-up may surface clear signs of depression.

A human staff member catches these because they're listening to a person, not processing a request. An AI agent only catches them if you build detection for them explicitly — as a listener running across the whole conversation, independent of whatever workflow the call started in, with its own escalation path that overrides everything else.

This is table stakes, not an advanced feature. Puppeteer's agents run safety detection for self-harm, substance abuse, and depression indicators across every interaction regardless of use case, precisely because these disclosures don't respect workflow boundaries. If a vendor treats safety detection as a triage-only module, ask why.

Guardrail 4: Make every decision auditable after the fact

When a triage decision is questioned — by a patient, a clinician, or a malpractice carrier — you need to reconstruct exactly what happened. What did the patient say? What did the agent ask? Which protocol fired? What disposition was given, and on what basis?

That means full transcripts, timestamped decision points, and retention policies your compliance team actually signed off on. It also means someone reviews a sample of calls every week during the first months of deployment — not because the system is expected to fail, but because reviewing calls is how you find the fifteen edge cases nobody thought of during configuration.

Practically: encryption at rest and in transit, detailed audit logs for every interaction, configurable retention, and the ability to pull up any agent–patient conversation on demand. If you can't produce a specific call from six months ago in under a minute, you don't have an audit trail.

Guardrail 5: Know which side of the regulatory line you're on

The regulatory picture moved in January 2026, when the FDA issued updated final guidance on clinical decision support software. The headline change was deregulatory: FDA will now exercise enforcement discretion for CDS that gives a single directive recommendation in situations where only one option is clinically appropriate — software that would previously have been pushed into device territory for not offering the clinician a set of options.

Time-critical decision support got reorganized rather than freed. FDA moved that analysis under the "can the clinician independently review the basis?" criterion, reasoning that urgency increases automation bias because there isn't time to weigh other information. Software built for time-critical decisions still generally lands on the device side of the line — the guidance's own example of a tool predicting cardiovascular events within 24 hours remains a device.

The detail that matters most for a triage deployment is who the software talks to. The non-device CDS pathway is written around software that supports a healthcare professional who can independently review the basis for its recommendation. FDA is direct about the converse: software intended to provide recommendations to a patient fails that criterion and meets the device definition. A patient-facing agent that issues a disposition on its own is a meaningfully different product, and the exemption logic does not transfer to it.

The practical read: keeping a clinician in the loop on anything close to a clinical determination isn't only a safety choice, it's a regulatory posture. Have your counsel confirm where your specific configuration sits before launch — this is guidance to bring to that conversation, not a substitute for it.

What to check in your first 30 days

Once the agent is live, four numbers tell you almost everything:

1. Escalation rate by tier — and whether it's drifting. A sudden drop in urgent routing is a warning sign, not a win.
2. Handoff rate and handoff reason — clustered reasons point at a protocol gap you can fix.
3. Time from red flag to human contact — measured in minutes, not hours.
4. Clinician agreement on sampled calls — have a nurse review a weekly sample and mark whether they'd have routed the same way. Anything below the high nineties needs configuration work before you expand scope.

A triage agent that never escalates isn't efficient. It's broken, and you won't find out from a dashboard that only counts calls handled.

Triage automation works when it's narrow, protocol-driven, and honest about its limits. The goal isn't an agent that decides — it's an agent that never lets a 4:50 p.m. call go to voicemail, and always knows exactly when to put a human on the line.

Ready to build one around your protocols? Let's build your next care agent together →