I’m building out a new voice agent system for a short engagement with a client, and I’m deliberately focusing on complex, embodied use cases. There’s an aviation focused agent I’m working on, but I can’t afford to buy a plane, and there’s only so much you can learn from a sandbox before it starts lying to you.
So I bought a cheap surplus military fleet truck. A 20-year-old Chevy Avalanche for next to nothing. It’s not my primary vehicle, so nobody at home is going to be upset about me tearing into the wiring. It’s one of those mostly mechanical Chevys that runs forever and doesn’t phone home with a ton of needless telemetry.
The idea is that a lot of the interesting pieces here are structurally universal. Routes, internals, a flexible GPS layer, and most importantly... the ability to add and remove sensors and see how robust the core framework actually is under change.
Memory in the embodied system
The other piece I’m excited about is context management. The trick is building the system around a custom model that’s strong enough to use the memory system unprompted, rather than treating memory as a retrieval crutch.
The model of choice is a fine-tune of Gemma 27B with multimodal capabilities. It’s a clever model with the essentials baked in. It has voice, image processing, and an MOE architecture that makes it viable in a quantized configuration, streaming tokens fast enough that video isn’t strictly necessary. I’d care more about video if I were chasing full self-driving, but here I’m more interested in LIDAR differentials and perception deltas.
Unlike planes or boats, road vehicles are interesting because of how many rules they force you to reason about. As long as the thing moves, I can still look up key locations, react to signage, and listen to radio signals. It’s a surprisingly rich constraint space.
Speech, change and local resources
The biggest pain point so far has been the pace of change in TTS. It’s exciting to see so much progress in open-source speech, but it’s also disorienting to finish a working conversational stack only to find that NVIDIA or Microsoft has dropped new libraries three days later that obsolete parts of it. This has already happened twice.
The upside is that the framework itself is flexible. At this point, adapting mostly means retraining the tooling LoRA rather than rethinking the system.
There’s also the resource side: doing all of this without the cloud. It finally feels like the hardware ecosystem is mature enough to make this practical, with affordable options that can survive quite happily on a 12V system.
I’m looking forward to messing with antennas, installing cameras, and doing actual wiring.
This is going to be fun.
I can’t wait to see where this breaks.