dimamitya

← writing

teaching a voice agent to stay quiet

in a shared voice room, every utterance is a possible input. only some of them give the agent a reason to speak.

consider a simple example: claudima asks a question, another person asks something else, and someone says “yes.” that answer might belong to the human conversation. remembering that the agent recently asked a question is insufficient to claim the reply.

this became a concrete problem in claudima-clubhouse. monitoring caught replies to praise and to people discussing the assistant in the third person. the application needed a better account of who was addressing whom.

how a continuation window went wrong

the conversation runtime used timed windows to let people continue without repeating the assistant’s name. one window tracked the last interlocutor; another admitted possible continuations from the room.

an audit found that eligibility was leaking into the decision to answer. generation received an utterance already labelled as a continuation. new assistant speech extended the window again. a question mark acted as a weak proxy for expecting a reply. even a silent action or a text-only response could open the shared voice window.

there was a second problem: another participant’s candidate utterance could take focus and cancel an answer already being prepared, before the system had decided whether that participant was addressing the agent.

changing the timeout would have left those state transitions intact. i made the distinction explicit: a candidate enters routing; routing decides whether the conversation calls for a response.

give waiting and silence separate meanings

the context-routing path has three outcomes:

the router considers recent speech, changes of addressee, the last delivered response, and active conversational context. it can still get the meaning wrong. the application therefore keeps control of the consequences: an ignored candidate does not take over another person’s pending request, and silence does not extend the conversation window.

permission to speak remains separate from permission to act. a routing result does not grant moderator rights, change the target account, or make a stale room connection usable. those checks belong to the application.

measure the interruptions

i built a synthetic corpus from 80 base situations, each with five variants: the original, missing punctuation, repetitions and filler words, intervening speech from other accounts, and a distorted name. the checks use the real rust context builder.

348 inputs had a usable expected label. the remaining 52 were marked ambiguous. when a damaged name could refer to either the assistant or a person, i kept the model’s prediction without pretending the hidden intention was observable.

one recorded run of the experimental local gemma4:e2b-mlx-bf16 router produced:

check result
correct labelled decisions 245 / 348 (70.4%)
answers when silence or waiting was expected 67 / 184
missed answers 7 / 164

the model answered in more than a third of the cases where it should have waited or stayed quiet. it was much more willing to enter a conversation than to miss a chance to respond. that error distribution matters to everyone sharing the room.

routing was fast in this run: median 155 ms, p95 213 ms across the 400 inputs. those timings cover the routing decision only. speech recognition, response generation, synthesis, and playback are outside them. the experimental router remained disabled by default after this evaluation.

keep the evaluation honest

the five variants share a base situation. splitting those variants across development and evaluation would make the two sets less independent than they look. i also tracked whether an entire family retained the right behavior through its unambiguous variations.

these are developer-labelled synthetic cases. they reveal particular failure modes; they do not estimate accuracy in arbitrary live rooms. the text distortions exercise some consequences of recognition errors without measuring the recognizer itself.

session tests cover another layer: interruption, cancellation, expiry, and room changes. connecting those checks into a reproducible audio-to-decision run is further work. a correct label on a prepared transcript cannot establish the behavior of that whole sequence.

for this agent, being useful includes leaving space for other people. the evaluation needs to count the times it takes that space unnecessarily, and the runtime needs a first-class way to wait or stay quiet.

also in this project: building the clubhouse client and testing recovery by breaking the connection.