An apparatus includes a plurality of processing layers coupled in series. Each processing layer in the plurality of processing layers includes a Gaussian unit configured to perform a linear transformation on an input signal including a plurality of optical modes. The Gaussian unit includes a network of interconnected beamsplitters and phase shifters and a plurality of squeezers operatively coupled to the network of interconnected beamsplitters and phase shifters. Each processing layer also includes a plurality of nonlinear gates operatively coupled to the Gaussian unit and configured to perform a nonlinear transformation on the plurality of optical modes. The apparatus also includes a controller operatively coupled to the plurality of processing layers and configured to control a setting of the plurality of processing layers.