Systems and methods are described herein for enabling, on a local device, a voice control system that limits the amount of data needed to be transmitted to a remote server. A data structure is built at the local device to support a local speech-to-text model by receiving a query and transmitting, to a remote server over a communication network, a request for a speech-to-text transcription of the query. The transcription is received from the remote server and stored in the data structure at the local device in association with an audio clip of the query. Metadata describing the query is used to train the local speech-to-text model to recognize future instances of the query.