A system to generate a space-time scattering network includes a computing unit configured to store a space-time scattering network (STSN) algorithm. The computing unit also executes the STSN algorithm to represent scattering of light through a material mathematically as a tensor field at a location. The computing unit also generates a subnet that represents the tensor field at the location. The computing unit also identifies, based on a translation operator, connections between a plurality of subnets, where each of the subnets in the plurality of subnets represents a given tensor field at a given location. The computing unit also forms the STSN based at least in part on the identified connections between the plurality of subnets. The computing unit further uses the STSN to inversely design an optical component or to perform tomography.