A system and method for directional sound sensing directs an optical sensing beam to an object of interest having a rough surface that vibrates acoustically. The light is reflected thereby and scattered as a speckle pattern that includes multiple speckles having a random distribution of phase offsets. A detector array having multiple detector elements receives and detects the speckle pattern and produces signals that are linearly proportional to phase modulation of the speckles. A summer receives signals from at least two of the detector elements that are offset at different phases and sums the received signals to generate a non-vanishing signal representative of an acoustic signal.