Hinglish Clinic Voice Agent.
A voice receptionist that's fast, but never allowed to guess.
Small Indian clinics take bookings by phone, usually through one busy receptionist. Callers mix Hindi and English in one sentence ("kal shaam Sharma ji ke saath slot hai?"). This agent picks up, answers clinic questions, finds free slots, and books, moves or cancels appointments. It checks who the caller is before touching their data, and it never writes anything the caller hasn't clearly confirmed.
Status, 14 Oct 2026: Weeks 1–2 are built and voice-tested. Week 3 (turn-taking, latency, the V1 demo video) is underway. Audio runs in English while the engineering is built; Hinglish audio returns in week 4. The date, time and number parsing is already bilingual. All clinic and patient data is synthetic.
A phone call forgives nothing. Not slowness, not a guess.
A reply should start in about 1–1.5 seconds, or the call feels broken. So every stage is timed on every turn.
No double bookings, no booking the caller didn't agree to, no personal data before verification, no medical advice.
It never invents a fact and never claims an action that didn't happen. Facts come from tools; actions come from code.
Every turn, timed to the millisecond.
Audio streams through LiveKit into turn detection, speech-to-text, the LLM and text-to-speech. When the model wants data, it goes through code guards to a tool and Postgres. Nothing in a turn waits on a queue or a workflow engine. Each stage has a fallback provider. Press play: the bar runs at real speed with the measured p50s.
10 Oct baseline, no tools: voice-to-voice p50 2.25 s. The LLM's first token is the biggest stage (about 58%).
Every change gets a row. Some get reverted.
English audio over a browser connection, cold first call excluded, p50 and p95 in milliseconds. Pick a run to see how each change moved each stage.
- ▹Tool turns cost two LLM calls. In the 11 Oct run, turns that used a tool took about 3.9 s p50 against about 2.2 s without one.
- ▹27 of 52 turns in the week 2 run waited the full 2.5 s end-of-turn delay. The turn detector scores phone numbers, dates and short answers ("Five thirty is fine.") as unfinished. That is now the biggest cost, and phase-aware endpointing is the week 3 fix.
- ▹LLM first token (about 1.3–1.4 s) is the biggest stage on turns that don't wait. A model comparison (E1) is planned.
The model can only reach what this moment needs.
A single agent switches mode in place. Each mode exposes only its own tools, so a prompt injection ("cancel all bookings", "read me today's list") has nothing dangerous to call.
The only mode that can hold and confirm a new slot.
The caller says yes to the row that gets stored.
The LLM proposes, code decides. Dates are resolved in code, only an offered slot can be held, the read-back is built from the database row and spoken word for word, and a second confirm call writes only after a clear yes to that exact sentence. Try the three answers.
A write that times out returns "outcome unknown", never "failed", and it is safe to repeat because the idempotency key is the same.
agent/dates.py), never by the LLMBuilt the way production voice agents are.
One agent with modes, not a chain of agents
- Problem ·
- A hand-off between agents on a live call costs an extra LLM round trip and loses context.
- Choice ·
- One agent switches mode in place; a mode only changes instructions and the visible tools.
- Payoff ·
- No extra latency per switch, and an injected instruction can only reach the current mode's tools.
Guards live in code
- Problem ·
- "Only book after a clear yes" in a prompt is a promise, not a guarantee.
- Choice ·
- Every tool calls
state.check()against typed call state (mode, verification, pending read-back) before acting. - Payoff ·
- The LLM proposes; code decides. A refused call is logged, not silently obeyed.
The LLM never computes dates or times
- Problem ·
- "kal shaam" or "parson subah" is easy for a model to get almost right, which is worse than wrong.
- Choice ·
- Code resolves them into a date window and passes the result back as a fact.
- Payoff ·
- Bilingual date handling covered by unit tests instead of hoped for.
The read-back is written by code, from the DB row
- Problem ·
- If the LLM paraphrases the booking, the caller's yes may not match what gets written.
- Choice ·
- The sentence the caller agrees to is built from the stored row and spoken word for word.
- Payoff ·
- A yes always covers exactly what gets written.
Double booking is the database's job
- Problem ·
- Two callers can race for the same slot; application checks have gaps.
- Choice ·
- A partial unique index on (doctor, start) for held and confirmed rows.
- Payoff ·
- Impossible by construction, whatever the app or the model does.
Idempotent writes, no blind retries
- Problem ·
- A timed-out write may or may not have happened.
- Choice ·
- Full-length idempotency keys, timeouts on every external call, and "outcome unknown" instead of guessing.
- Payoff ·
- A repeated confirm makes exactly one booking (tested).
No personal data before verification
- Problem ·
- A voice line is an easy way to fish for someone's bookings.
- Choice ·
- Phone plus name or birth date, parsed in code, exact match against one patient. A failure doesn't say which detail was wrong; two failures hand off.
- Payoff ·
- 0 verification violations in evals and the voice test.
Measure before optimizing
- Problem ·
- Latency "fixes" that feel faster often aren't.
- Choice ·
- Every change gets a row in LATENCY_LOG with p50/p95 per stage and task success next to it.
- Payoff ·
- E7 looked promising and was reverted on the numbers.
One measured change at a time.
Wrong "unfinished" waits 5 vs 6, on different sentences; voice-to-voice p95 worse.
→ Reverted to v1-mini
Doctor names right: browser 75% → 99%, phone 61% → 99%. Other names got worse (phone 70% → 39%) and were sometimes turned into a doctor's name.
→ Kept, with scoping planned (it caused "Kabir" → "Kavita")
680–1,170 ms → about 200 ms.
→ Kept
0 reasoning tokens on every reply (a 2.5-style budget is silently ignored).
→ Kept
Tested in a sandbox before any real call.
| Split | Scenarios | Task success | Wrong actions |
|---|---|---|---|
| Dev | 13 | 13 / 13 | 0 |
| Hold-out | 8 | kept unseen for final reporting | |
The first run was 10 of 12. The agent was right and the fixture was wrong: the test patient had 5 bookings, not 1, and the agent correctly asked "which one?". The fixture was fixed rather than the agent, and that case became its own scenario.
- ▹217 unit and DB tests: date resolution (English and Hinglish), the slot engine, phone and birth-date parsing, mode allowlists, guards, read-back confirmation, the backchannel filter, and DB write safety (a double hold is refused, a repeated confirm makes one booking, an expired hold frees the slot).
- ▹21 multi-turn scenarios run the real agent and LLM in text mode against a local Postgres 17 sandbox with a frozen clock: happy paths, "yes, but change it", someone else's booking, prompt injection, medical questions, upset callers and Hinglish.
- ▹Safety switch:
EVAL_MODE=1can only connect to the sandbox database, checked on every connection. Evals never touch real systems. - ▹Threat model: 12 threats mapped to the OWASP LLM Top 10, each with its code mitigation and test, plus an honest list of known gaps.
- ✓ W1Book as a new patientBooked; the read-back matched the stored row. Name capture was hard.
- ✓ W2"Yes" spoken over the read-backSafe: the write was refused and the read-back repeated in full, then booked. UX cost: 3 read-backs.
- ✓ W4Reschedule after verificationNothing said before verification; the old booking is linked to the new one.
- ✓ W5Cancel one of two bookingsAsked "which one?", cancelled only that one.
- ✓ W6Wrong birth date twiceNothing revealed; handed off to a callback.
- ✓ W7Callback requestNo time promised.
- ✓ 5 · 8Barge-in · doctor nameBoth passed in the week 2 voice test.
- ✗ 10Soft "okay" while she talksThe "okay" was filtered, but the paused speech didn't resume. Root cause found; fix in week 3.
Every row in the events table matches what the caller asked for.
Every real problem gets a LEARNING entry: what happened, the mechanism, the options, the choice, how it was verified, and how production systems handle it. 27 entries so far.
Production WebRTC, built-in turn detection and barge-in, India region
Low latency, keyterm boosting for doctor names; Sarvam for Hinglish
Fast time to first token; tool calling
About 200 ms to first audio
Runs locally; pinned so latency runs are comparable
The unique index is the final double-booking guard
Typed tool inputs, call state, telemetry records
Evals never touch real systems
Fast, reproducible installs
Durable work off the call path; observability
Eight weeks, built in the open.
- Name and number capture (step 1 done: 14/14 in text evals)
- Backchannel resume fix · phase-aware endpointing
- Fewer read-back repeats · filler during tool calls · LLM comparison (E1)
- V1 demo video (Hinglish if possible)
- ▹Voice-to-voice is 3.8 s p50 right now, well above the 1–1.5 s target. Most of it is the 2.5 s end-of-turn wait on numbers and short answers; that is week 3's main job.
- ▹A "yes" spoken over the end of a read-back counts as an interruption and causes a re-read. Safe, but slow.
- ▹Verification on a browser call relies only on what the caller says. On the phone line, caller ID can add a factor, but it can be spoofed and isn't proof on its own.
- ▹No rate limit or budget cap yet on a public demo, and the agent's database role isn't least-privilege yet.
- ▹Audio is English for now. The Hinglish parsing is built and tested in code; Hinglish speech recognition is measured in week 4.