Consumer AI hardware · Case study
A multimodal assistant that knew when to stay local.
A real-time assistant platform brought audio, video, multiple AI models, and home automation tools together on an NVIDIA Jetson—then used an on-device classifier to decide when a request belonged at the edge and when it needed the cloud.
- Work
- AI infrastructure and product engineering
- System
- Python, NVIDIA Jetson, edge and cloud inference
- Focus
- Multimodal pipelines, routing, tool access
- 01Audio + videoLive device inputs
- 02NormalizePython media pipeline
- 03Classify + routeDecide at the edge
The challenge
One assistant, very different kinds of work.
Real-time multimodal systems are not one model behind one API. Audio and video arrive on different clocks. Some requests are simple enough to resolve on the device. Others need a larger model, broader context, or a remote service. Tool use adds another boundary: understanding a request is different from being allowed to act on a home.
The engineering problem was to make those pieces behave like one product without forcing every interaction through the most expensive or distant path.
What the work involved
The software around the models.
The models mattered. The coordination layer made them usable.
Built the real-time media path.
Python services captured, normalized, and prepared audio and video for several inference paths while keeping the device constraints visible to the rest of the system.
Put classification on the edge.
A lightweight routing layer classified each request and chose a local or cloud path. The decision could reflect capability and context instead of sending every interaction to the same model.
Turned model glue into a system boundary.
Adapters isolated model-specific inputs and outputs from the product workflow, making it possible to combine specialists without leaking every provider detail through the codebase.
Gave the assistant scoped ways to act.
MCP-style interfaces exposed home automation capabilities through a common tool layer. The assistant could reason about tools without treating an integration as unrestricted device access.
What transferred
The right model is often a route, not a brand.
Edge AI made the tradeoff concrete: latency, capability, network dependence, hardware limits, and action risk all belong in the routing decision. That same pattern now informs agent systems that choose among specialist models, deterministic software, and human review instead of asking one general model to do everything.