Day 8: I now have a fully tooled, talking agent that sounds realistic enough that I’m thinking it could fool me in a phone conversation. I mean, if I didn't know it was there.
It responds in real time, too. There’s a tiny delay while it generates responses, but it’s small enough to feel conversational rather than synthetic.
This build stacks Whisper and Chatterbox across three generations of experimental agent frameworks. I finally took the time to consolidate the redundant pieces into something closer to a real product. The time pressure helped.
There’s nothing like a deadline to force architectural honesty.
Production notes:
Early on, I ran into complexity issues. The failure point was function calling, and it showed up across two different coding agents (Codex and GLM 4.7).
The problem wasn't model specific. It might've been me. It could've been the fact that I pulled together multiple older versions of Busy / Woven / nMF to save time and asked the AI to reconcile all of it.
Went great for a little. Record productivity! Then brain melt on days four and five.
But eventually it clicked.
Data structures are locked in, phonetic alphabet handling works, and all the pieces now cooperate like perfect little princesses!
I also redesigned the UI to support this setup. That part was fun. I don't mind UI development if nobody's asking me to write JavaScript by hand. AI abstracts the evil away.
The model and voice layer
For the model, I went with Qwen3-0.6B, 16k context window, with three layers offloaded to CPU.
People underestimate this model.
Give it strong base templates, disciplined prompting, and command training, and it’s a spry little trooper. Not the brightest bulb... but a surprisingly good actor.
The voice layer was interesting too.
I used Chatterbox, but downsampled TTS to 16-bit inference. You can’t really hear the difference, and reflex time is nearly instant.
Architecturally, the system now has two classes of agents:
An actor decides on its own tooling, and carries its loops, makes decisions, self corrects, and reports back to the user. Actors are stateful, and come with an overpowered memory system that keeps them coherent.
“Smart functions,” are boring - prompt in, result out. Conventional, necessary, unromantic.
Think subjects/objects = actors/functions. Same idea. At any given time, the model will be asked to play both roles.
I skipped the image-reading stack for now due to resource constraints. One of the requirements this round was that the whole system fit on a cheap Jetson board, so everything is tight.
I'm excited to show this one off next week.