A computer-implemented method is programmed to support efficient and rapid generation of machine translation suggestions on client devices. Network latency is substantially reduced or eliminated by separating certain aspects of the translation workload across multiple classes of tasks, including final neural network output, between a client device and server device. The client device and server device may be connected such that a decoder portion of a machine translation system may be downloaded onto the client device, along with an initial translation suggestion and encoder outputs associated with a document, which document is in a source language to be translated into a target language. The initial translation suggestion may be replaced by an updated machine translation suggestion as a user inputs text in the target language called a prefix. This updated machine translation is generated on the client-side decoder using the previously-downloaded encoder outputs as input and the prefix as constraint.